Nikolay: Hello, hello, this is
PostgresFM, your favorite podcast

about Postgres.

And my name is Nikolay.

My co-host is Michael.

Michael: Hello, Nikolay.

Nikolay: Hi, Michael.

And we have a very good guest today,
Sai.

Sai: Hey, Nik and Michael.

Nikolay: Founder of PeerDB.

We are very glad to have you here.

And we just discussed that we are
not inviting guests just for

guests.

We choose topic first, so it's
topic-centric discussion.

But my first thought when we discussed
we need to talk about

logical.

And also I saw your blog post published
recently.

My first thought was you're a very
great person to have for this

discussion, right?

So that's why we invited you.

Glad you found time to join.

Sai: Thanks Nik and Michael for
inviting me.

And yeah, logical replication and
logical decoding has been my

life since the past like 7, 8 months.

And we are trying to get into as
much depth as possible to, you

know, understand how it works and
probably down the line contribute

upstream as well.

But yeah, I'm very excited to be
here.

Nikolay: Cool.

I know Michael has a lot of questions,
so I probably will be

less active this episode.

But I will have questions as well
and maybe comments.

Michael: Yeah, I definitely have
questions.

But yeah, I completely echo everything
Nikolay said.

And yeah, we did a kind of a basics
intro episode to logical,

where Nikolay also talked us through
how to handle scale in terms

of major upgrades, at least.

So that's a common use case for
logical.

But 1 thing we didn't even touch
on, which was the topic of 1

of your recent blog posts, was
I listened back and I think protocol

versions got a single mention,
kind of like an off-the-cuff mention

at 1 point in the episode, you
know, 30, 40-minute episode.

So that was where I was thinking
of starting is, it'd be awesome

to hear a little bit about the
protocol versions, why you worry

about them recently.

Sai: Yeah, absolutely.

Thanks, Michael.

So the thing is logical replication
has this concept of protocol

versions that you could specify
as a part of the subscription

or if you are like an external
client who is reading the replication

slot you could do it as a part
of the start replication API which

lets you read the replication slot
and there are like four versions

that Postgres supports now with
PostgreSQL 16, right.

The first version is the default
version which decodes transactions

that are only committed so it doesn't
like start the decoding

process before the transaction
commits and the second is the

more advanced option which does
logical decoding of transactions

that are not yet committed, right
like so it basically does decoding

for in-flight transactions, right
that is number two and then third

is it lets you decode two-phase
commit transactions, that is third

and the fourth is it lets you decode
in-flight transactions and

apply them in parallel to the target
which is the subscriber

which is PostgreSQL basically, right.

Now in that blog which you're talking
about we compared like

version one and version two because
they are more like the common

ones.

Two-phase commits are not like
very common, right that's the reason

we didn't go into that piece but
the biggest difference is the

impact of two is it improves logical
decoding performance because

you are not you're actually decoding
while the transaction is

happening, right like you are not
letting the transaction finish

and then only start decoding but
rather you're decoding as the

transactions go on.

So the idea is you're giving more
time to the decoder, the WAL

sender process to perform the decoding
and now this has a lot

of benefits, right like number one
it reduces the slot growth, right

like so in that blog we talk about
a situation where we have

two long running transactions that
are being committed.

And with version one, the decoding
process doesn't start and the

slot keeps growing, growing, growing.

And only after the transactions
are committed, the decoding starts.

And then for a few minutes until
the decoding is finished, the

slot size remains the same.

Right now, this is with one.

With two, what happens is as the
transactions happen, decoding

starts.

Right?

Like, so you do see the decoder
doing its job.

And as in when the transactions
finish, the slot immediately

falls.

And the advantage of this is so
the summary was with version

one, the slot growth of like, I think
we showed like 5, 6 gigabytes

remain for like 5, 6 minutes until
the entire decoding finished.

But with version two, it remained
there for an instance, because

you know, like the decoding was
already happening as the transaction

was going on, right so this can
have tremendous impact in use

cases which have long running or
sizable transactions and that

is very common, right like we
recently working were working

with like a fintech customer, right
like where they had a lot

they had like sizable transactions
right like hundreds of like

you know operations happening in
a transaction and then these

transactions were like interleaved
and now the advantage with

like to the two would have helped
them a lot because as the you

know operations are happening the
decoder like you know just

decodes them.

With one the problem that was happening
was there was one large

long-running transaction which
took like an hour and then every

time the decoder needs to decode
committed transactions it was

decoding this long-running transaction
also so so the order of

time is more quadratic basically
with like a version one right

because you know wall is like sequential
right like so you keep

writing writing writing and then
like and then as and when there

is a commit like the decoder like
starts like working now the

problem is with version one as the
existing like the long-running

transaction is not getting decoded
still the decoder process

decodes it for like other transactions,
right?

So but with like, you know oh this
problem wouldn't have happened

because once this long-running
transaction is committed I mean

it's already consumed it's already
decoded if that makes sense

right so the high-level summary
is version two is very helpful

when there are like, you know,
long-running transactions that

are like, you know, interleaved.

Nikolay: So the lesson is upgrade
guys, right?

Just use the latest PostgreSQL version.

Sai: Version two comes with PG 14
version four is in PG 16.

Nikolay: Sorry.

I meant a small thing.

I've noticed you mentioned a subscriber,
which is PostgreSQL.

This phrase, attracted my attention.

It's not always PostgreSQL.

But parallelization which to apply
to long transactions it's done

only for PostgreSQL subscribers,
not for others, right?

Sai: Correct.

Exactly.

So the fourth version, which does
like, you know, parallel apply

of these in-flight transactions
is more relevant for PostgreSQL

to PostgreSQL replication.

The apply piece is, you know, PostgreSQL
and that's the standard

logical replication.

And logical decoding is more like
a subset of logical replication

where external clients like pg_BADGER
can read the slot and replicate

to homogeneous or heterogeneous
targets.

Nikolay: In your experience, does
it mean that there's no sense

to have this feature for other
situations like to Snowflake or

others.

Sai: The parallel apply feature
you mean,

Nikolay: right?

Right.

Sai: Yeah.

Good question.

So that is the onus of the ETL
tool.

So in PADB we have an option to
do parallel apply or not do parallel

apply.

So it's like a single flag that
we provide.

And the difference is with parallel
apply, we cannot guarantee

the order of transactions across
tables.

Nikolay: So foreign key and referential
consistency, similar

to multiple slots in regular Postgres
to Postgres.

Correct.

Sai: Very, very similar.

Right.

And then both, I mean, customers
do it, but the advantage of

parallelism is it will improve
latency, right?

Like replication, latency reduces,
lag reduces, etc.

But it doesn't guarantee, you know,
consistency across tables.

But the sequential does that, right?

So customers like we have both
customers.

Nikolay: Yeah, that's interesting.

I know like in many cases people
think, okay, we work at like

half of our capacity, no worries
if TPS grows using a single slot.

If TPS grows, we know the bottleneck
is on the subscriber, we will

just use multiple slots.

But then they realize that with foreign
keys, you need to basically

agree that they are broken temporarily
on the subscriber and

can cause a lot of troubles if
you point the application to such

nodes, right?

That's interesting.

Yeah, absolutely.

Makes sense.

Sai: And then I did want to call
out, right?

Like for that, I mean, there is
a setting for disabling foreign

key just like, you know, complete
that there is a setting called

session replication role basically,
you can set that as replication

and that is pretty standard, it's
like that's the reason Postgres

gave this setting so you can set
that to replication and it would

disable foreign keys and triggers
on the target.

Nikolay: But by default,
if you use multiple slots, it's

disabled.

Sai: By default, is it like disabled?

Nikolay: Yes.

I know it since recent work to
use it for upgrade, zero-downtime

upgrades.

Michael: Got it.

Nikolay: It's indeed disabled and
it's interesting.

So you need to think maybe you
shouldn't use it.

I mean maybe go back to a single
slot and just choose a different

time for your workload.

I mean, work on weekends, guys,
right?

Lower traffic and you can afford
working with a single slot.

This means only for like logical
for a temporary time, like for

upgrades, we don't need it.

Exactly.

It really

Sai: depends on the use case.

Nikolay: Yeah, yeah.

But if for a long time, forget
about foreign keys maybe.

And you said latency, but it's
also about throughput.

We need to process more bytes per
time.

Sai: Absolutely.

Nikolay: Lags can be nasty.

I don't like dealing with logical
decoding in Postgres 11, 12.

It's not pleasant at all.

So these improvements are exciting
and everyone should upgrade.

That's what I see here. You just
uncovered a lot of details.

That's great.

Sai: Yeah, absolutely. And the tricky
thing I mean with the good

beauty of like Postgres is
that like this is already inbuilt

in like Postgres core, right?

Like logical replication does
this by default.

It's just like, you know, a setting
you need to do say that like,

Hey, I want to use version 2 and
it will be significantly more

performant, reduce replication slot growth.

But if it's an ETL tool, who's
using logical decoding, the transaction

logic needs to be managed by the
ETL tool.

Because as you're reading like
in-flight transactions, we need

to keep track whether this transaction
is committed or not, and

then only we should push it to
the target.

It becomes tricky for ETL tools,
which we are working on in PADB

now as we speak.

But the beauty of Postgres is that
it just gives you the setting

out of the box and you just need
to upgrade your Postgres version

to 14, 15, and 16.

Nikolay: And with this, in 16 we
have an example.

I mean, the developers of these
tools like your company, they

now have an example of Postgres to Postgres
native logical replication to show how

it can be implemented.

So the reference example exists.

So it's good.

Yeah, that's great.

Sorry if my client interrupted
you.

No, not at all.

Michael: This is a really good
diversion.

But we've also, we've talked about
3 things already, right?

We've talked about throughput,
we've talked about lag, and we've

talked about slot size growth.

And I feel like when you said initially,
the performance of the

version 2 protocol can be better,
I'm guessing we're talking

mostly in terms of that lag metric.

Like we can start processing things
faster, therefore the lag

is lower, makes total sense.

And when you said the slot size
growth is reduced, I think I

Understand where you're coming
from.

I think the specific benchmark
in the blog post shows that the

growth is the same, but then reduces
it, like it kind of reduces

quicker.

But when you mentioned the like
interleaved long running transactions

I guess it's in a less synthetic
workload where you've got lots

of potentially overlapping long
running transactions the peak

will be lower in the version if
you're using.

Correct.

Sai: Exactly right. Like because
in the scenario we did it's like

just do 2 transactions that are
long running and then we just

had 2 of them right but then and
then they ended at the same

time but in real-world workloads
like it I mean it can be very

arbitrary right but with like version
2 as the slot consumption

is faster and like slot size falls
quicker right the cumulative

effect can be very significant
in slot growth.

Michael: Nice.

Sai: If that makes sense. And that
is the next benchmark we want

to do, where in the real world benchmark,
I want to see how does

the peak slot size like compare
with version 1 and version 2.

We are actually building that feature
so we can we plan to like,

you know, we have a few like customers
we are design partnering

with, you know, to implement this
feature.

And we have like, we know, you
know, what is the peak size slot

size we are seeing with version
1.

And with this design partnership,
we will get to know, okay,

what is the peak slot size?

My like understanding is it will
fall very quickly because like,

you know, you're giving more time
to the decoder, right?

Like it's more efficient.

And with these long running transactions,
it cannot be quadratic,

like the decoding process, right?

But that's a very good question
you pointed, Mike.

And that is the next benchmark
we want to do.

Michael: Nice.

So then the question then becomes,
are there any workloads?

And I'm thinking now that it must
only be if you had, let's say

you had a lot of long run transactions
that end up getting aborted

or rolled back instead of committed.

I'm thinking that might be the
only case where the version 1

protocol might be like you might
prefer to carry on using it

or are there other cases where
you might prefer to carry on using

the v1 protocol over the v2 one
as long as you're on 14 and above.

Sai: Even when the transactions
are rolled back the WAL is not

removed.

So it still needs to be like decoded
basically, right?

Like for future transactions.

That's exactly what happened in
a customer scenario where there

was a long running transaction,
which the customer killed basically.

It was running for like few hours,
right?

But still that impacted other transactions
because Postgres is

not yet smart where it removes
the WAL.

Nikolay: This is my favorite use
case.

So you can overload WAL sender.

If you just do a very simple trick,
you create a table with like

a million, 10,000,000 rows and then
delete rows in transaction

and roll it back.

Massive delete rollback, massive
delete rollback.

I like this workload in many cases,
this included.

And this makes WAL sender consume
100% CPU very quickly.

Because this transaction spams
WAL, writing xmax constantly

for a lot of tuples.

And then saying, okay, this xmax,
this transaction ID was rolled

back, so it doesn't mean...

Like they're still alive, right?

The next transaction comes, does
the same, and the WAL sender

is becoming crazy.

And I think there should be some
way to optimize decoding to

understand it, maybe.

Because if you start processing...

Ah, when it starts processing,
it's not known that this transaction

is already rolled back.

But maybe there is some opportunity
to optimize because if it's

known already it's already rolled
back why we can probably skip

better or something I don't know
I don't know details here.

Sai: That's a very good point Nik
and I was recently chatting

with one of the committers and we
don't even like skip rows is

my understanding basically like
so every time like we try to

decode basically So the immediate
optimization is if there is

an entry in the WAL of a rollback
transaction, then we don't

decode that entry in the WAL.

But you bring up a great point
here.

So there is a lot of scope of improvement
there.

Nikolay: Yeah, and this led us
to the conclusion that we won't

be able to deal with logical at
a few modifying transactions

per second.

It was wrong conclusion because
this was pathological workload.

And In production, I think it's
a very rare case when a WAL

sender is hitting 100% of a single
core.

Or it's not, I'm not sure.

Because the problem is you cannot
scale it.

You add a second slot, but you
see a WAL sender is hitting 100%

of a single core CPU, with the
constantly or from time to time,

and you think, okay, I will try
to distribute workload among

2 slots or 4 slots, 8 slots.

But this doesn't help.

All of them are hitting 100% of
different cores and that's it.

So this can be bottleneck, but
it looks like it's very rare right

now in production.

Sai: Yeah, absolutely.

And that is another optimization
that like, I mean, the community

could do is where we, I mean, currently
if there are multiple

slots, right?

Like the decoding process runs
for every slot, right?

If there are ways in which the
decoder runs for just 1 slot and

the other slots reuse these like
decoded changes, right?

Like if that makes sense.

Nikolay: Yeah, exactly.

The same work that's done multiple
times.

Everyone is processing everything.

This is what like definitely there
is opportunity for improvement

here.

But I think people just don't see
this as a bottleneck in production

often.

That's why it's not optimized yet.

Sai: In production, what we see
is like, it's like these 1 off

batch operations where customers
delete like a bunch of data

or add like do copy with like millions
of rows where there is

like a WAL spike.

And you know, the recommendation
that we give right, like is

guys have more disk.

I mean, it is, I mean that it's
hard because like faster and

like, you know, larger disk because
logical replication can

only scale up to 25,000 to 30,000
like, you know, messages per

second, basically.

So have larger disk so that like
once this small spike like falls,

it'll catch up.

Okay, sure.

At that time, there is more latency.

Nikolay: It's a good CPU if 25,000,
30,000 messages per second.

Because for PgBouncer on modern
Intel and AMD, I didn't see more

than 20 messages per second.

We can compare this like also messages
per second.

PgBouncer processes transactions
just by passing them to backends,

Postgres backends and returning
the result.

And the logical replication also
similar, some messages, just

some messages, right?

Different type of work because
decoding maybe is more consuming

but 30,000 sounds very good.

Sai: Yeah, 100%, but the thing is
it has to be done well, no I mean

like it has to be done properly
because one of the tips that

we do right, like which have seen
lack in a few ETL tools, is always

consuming the slot we cannot give
up on consuming the slot we

need to constantly consume the
slot and flush the slot right

and not to lag and the thing
is if you give up and start

reading it again, it doesn't read
from the confirmed flush LSN

it reads from the restart LSN.

And sometimes that restart LSN
can be very like old.

And I don't want to get into when
Postgres updates the restart

LSN because it's a more complicated
thing.

But the idea is when I give up the connection and reacquire the connection

again, it starts from the restart LSN, which can increase my

decoding times a lot.

So this is very common.

When we initially built pglogical, if the slot size was 150, 200 gigs,

we were giving up the connection periodically.

And whenever this slot is big, for like 10, 15 minutes, it was

just stuck in Walreader.

It was not even getting 1 change.

So that is one tip I recommend, where you always consume the replication

slot.

Nikolay: I was trying to recall another problem I saw when I

inspected what's happening if you start using multiple slots.

And it's kind of an interesting problem when you start publishing

changes to a single slot you use for all tables, for example.

For example, for upgrade, it makes sense.

And this is great.

It works very well.

But if you start distributing tables among multiple slots, you

need to alter the table, I think.

Anyway, you need a lock for each table individually in the group.

So you divide groups, and then you need to start publishing changes.

And this is a similar lock as, like DDL, basically.

No, not similar.

It's not an exclusive lock, but it's surely.

I remember it was, it got stuck.

It was not, it's not an exclusive lock.

Yeah.

So it's, it's not that bad as, as altering a table and adding a column.

So I remember it got stuck.

It couldn't complete, but at least it didn't block others.

Sai: Right.

And was this while creating the multiple slots, or was this just

like creating publications?

Nikolay: Creating a publication.

So you specify tables for which you need to create a publication.

And this needs to acquire locks on all of them.

It can be successful for some of them, not successful for others.

It's a single operation, so you're waiting.

And this is not good.

But not as bad as DDL, which can block selects, which come after.

We discussed many times how dangerous it can be under heavy load.

Yeah, that's interesting.

Anyway, multiple slots are tricky.

This is the lesson here.

So you need to test and learn before using them.

Michael: Yeah, absolutely.

Nikolay: Good.

What's next?

Michael: Well, before we move on from that, I'm interested in,

Are there any other like hard,
so you mentioned that rule of

thumb, 25 to 30,000 messages per
second.

Like, are there any other like
rules of thumb or hard limits

or anything that people could be
thinking, oh, we've probably

not tuned it enough because we're
not getting that much throughput

or that low lag or something like
that.

Nikolay: I guess this number depends
on the, on the core, on

the type of CPU you have.

If it's old, it can be 10,000 for
example.

Sai: Yeah.

A hundred percent.

And then, you know, this setting
is this number that I shared

is more on the higher end in a
non-local environment where the

source, the target are not local.

They are in the same region, but
in different boxes.

So network latency is a very big
factor.

1 thing we do want to see is, what
is the logical replication

performance when both are on the
same local server, which is

not in real world that doesn't
happen.

So this, whatever I'm saying, is
network latency bound.

Because it's single-threaded, and
the network latency kicks in,

and it can only scale up to say
20-30 Mbps if it's like you know

30-40 Mbps if it's done well also
right so those are also some

things to keep in mind now coming
to gotchas right like a few

things that I did like you know
make a note of of logical replication

first is it doesn't support like
replication of DDL commands

which includes like adding of columns
like dropping of columns

like you know adding new tables
creating indexes like you know

truncating tables right like so
these are not supported out of

the box.

And this is 1 of the common concerns
that we hear from customers,

because in the real world, you
know, people, I mean, you add

tables, you like you have these
Django migrations, which like

add a bunch of indexes etc etc
right.

Michael: Partitions we talked about
last time.

Sai: Yeah, but they don't do that
also.

Michael: So yeah, when you say
customers raise it as a concern,

is that before starting or like
realizing it later on?

Where does that pop up for you?

Sai: It happens both ways, right?

Like because I mean, obviously,
Postgres logical replication

is the easiest way to like, you
know, replicate databases, right?

They start off and then there's
a column that is added and then

logical replication breaks.

And now the good thing is you can
add a column manually on the

target and it would continue But
then this becomes like difficult

in production, which is when they
reach out saying that hey you

guys like are also, you know Doing
using logical decoding and

you support Postgres as a target
and that's how they come to

us.

And we do support schema changes,
but only add columns and drop

columns.

We don't yet support creating tables
and indexes and truncate

and all of that yet, basically.

And sometimes it happens before
itself.

We're like, okay, this is a, I
mean, we keep adding tables, we

keep like adding columns very often
and like this doesn't work

for us.

So like we want another solution.

Michael: Yeah, nice.

A couple of the other things that
I see people confused or struggling

with are things like sequence synchronization.

I guess that's just a one-off task
at the beginning.

Do you see anything else catching
people out?

Sai: That's a good question, right?

Like I think DDL commands is number
1, replication slot growth

issues is number 2, which we talked
a lot about basically.

Yeah.

It's like, and, you know, always
keep reading the slot, avoid

long running or like open transactions,
you know, monitor slot

growth, right?

Like fourth is, you know, use protocol
versions, right?

Like upgrade to like latest Postgres
versions and start using

the version 2.3.4 which is faster.

The third thing that we see is
Postgres logical replication doesn't

replicate toast columns out of
the box.

You need to set up replica identity
full to make that happen.

And replica identity full could
be expensive if you are not on

Postgres 16 because for updates
and deletes it could lead to

like a sequential scan.

Nikolay: And this thing what happened
CTID was used or is used

or like why it's better than 16?

Sai: It's using indexes now.

It's like using indexes basically.

Nikolay: Okay.

Sai: And the thing is,

Nikolay: If you have them.

Sai: Yeah, if you have them and
if you have the right indexes

And another thing is some customers,
right?

Like who have primary keys, but
they have toast columns.

We still like need to do replica
identity full and that helps.

So replica identity full with primary
keys is more efficient.

And there is a good blog that the
Zalando guys wrote, which

I really liked.

And that is something that I refer
to customers where, hey guys,

like you can add this.

I think it increased like CPU and
IO by 30%.

That was their analysis.

But that is about like toast columns
where you need to have replica

identity full and the impact of
that can change based on scenarios.

Michael: Yeah.

Awesome.

I didn't know about that one.

I would love to read that blog
post.

If you send it to me afterward,
I'll include it in the show

notes as well.

Sai: Absolutely.

And the fourth thing that we have
seen is logical replication,

at least, you know, logical decoding
did not support virtual

and generated columns.

If there are like generated columns,
they didn't show up on WAL

decoding.

And the way we, I mean, we easily
solve that, I mean, is on the

target that customers set up like
a virtual column or they use

like DBT or like some transformation
tool to, you know, backfill

this, right?

Like, so that is number four.

And then last but not the least,
the fifth one that we see, right.

Like, I'm still, you know, trying
to discover, it's like slot

invalidation issues where the restart
LSN becomes null, and have

seen that happen in two scenarios.

So the first scenario is there
is a setting called max_replication_slot

size or something, I mean, like which
actually limits the size

of the slot, and in this, this setting
was recently added in 13 or

14 which lets you like safeguard
from storage out of storage

issues. So as and when the slot
hits, you know, over a 2 terabyte,

right, like or if that's the setting
that you have, it basically

invalidates the slot, and in that
scenario, the restart LSN

becomes null, and the slot gets
invalidated, and you need to restart

logical replication.

The second scenarios we have seen
is sometimes it happens randomly,

also, unseen, and we are still
figuring out why that can happen.

And now we reached out to the community,
right, like, and we heard

that like it could be because of
cloud providers' Postgres, right,

like because cloud providers have
a bunch of like backup operations,

some, some forks, you know, of Postgres, right.

And then it could be. I mean, the
community doesn't say that,

like, I mean, their point is, like,
can we try to reproduce this

on Vanilla Postgres, right?

Like not cloud.

And then that was not becoming
easy for us.

But like, we have seen that out
of the 7 to 8 months, 1 to 2

times, randomly the slot gets invalidated
and we don't have like

an RCA on when that can happen.

And immediately we think that we
check, hey, is the max_wal_size

for the slot, like, less than, is
it set up, right?

But it's still set to minus 1, it's disabled,
but still, like we run

into these issues.

So I'm on that quest of figuring
out when this happens because

it's a scenario that we need to understand better.

Michael: And that's terrifying.

And yeah, how do you even begin
to go about reproducing that?

Good luck.

Sai: Exactly. So we were able to
reproduce this on one of the cloud

providers.

They're like for high throughputs,
like over 50K transactions

per second, every 15 to 16 hours,
it was like getting invalidated.

Right?

Michael: 50,000 per second and
every 15 to 16 hours.

Sai: It's a lot.

That's a lot

Michael: of messages.

Sai: That's not real.

I mean, I don't think that happens
often, but yeah.

Michael: Right, wow.

Cool.

I don't feel like we talked about
use cases much.

Is that because it kind of these
aren't use-case dependent?

I guess other than people using
two-phase commit, do you see any

differences between people that
are using logical replication

for analytics databases versus
for, do you see any difference

between use cases basically?

Sai: Good question.

Like I mean, logical replication
is very powerful.

I mean, it supports like, you know,
migration, online migrations,

number 1, then it supports like
online upgrades, right, like

which Nik mentioned, right, like
number 2.

Number 3, it also supports like
HA and backup scenarios.

So I have seen some customers who
use logical decoding or

logical replication to have HA
across regions or in like hybrid

environments.

Like, okay, I'm running stuff on
my on-cloud and I still want

to have a backup or like a replica
on my on-premise like, you

know, self-hosted environments,
right?

And the only way it's possible
is via logical replication because

cloud providers don't give access
to like WAL, right?

That is number 3, where like HA
and backups is another use case,

right?

4 is replication to like non-Postgres
like targets or even Postgres

targets for like workload isolation,
use-case isolation, where,

okay, I have my OLTP database running
on Postgres.

Now I want to pipe this to another
database.

Like it could be Snowflake or Postgres
for analytics or for search,

like I want to pipe this on like
a Clickhouse or like Elasticsearch,

right?

Like for optimizing search.

Yeah, logical replication is very
powerful and I believe that

it's going to be the future for
Postgres and down the line it

can open up like active-active
use-cases also right like where

you know, okay, you have like cross-region and I recently read

a blog where folks were using logical
replication for like active

active I'd like so I mean it opens
up like a plethora of use

cases and makes Postgres more,
you know, powerful extensible

and yeah

Michael: Yeah, I read that blog
post too.

I think it was version 16 change
that made it possible to like

not replicate changes that got
made by logical replication.

So you could kind of like set up
logical replication in both

directions.

It seems super dangerous to me.

Like, okay, cool.

Would you recommend it at the moment
or do you think we need

more features before that?

Sai: I think it's more, I mean,
it's more like intense than that,

right?

Because like conflict resolution
and all of this is tricky, right?

Like, and, you know, I mean, if
it's like very workload specific,

where like, okay, I don't touch
the same rows, right?

Like, and, you know, maybe there
it could work.

But I mean, out of the box, like
implementing that is tricky,

and it requires more like effort.

And maybe we go there in the future,
right?

Because we are seeing a few cases
with customers where they want

Active-Active and there is not
an out-of-the-box solution.

Nikolay: Why do people need Active-Active?

Sai: Good question.

I think I have like a lot of thoughts
here.

So like the thing is, I think it
really helps with like HA, right?

Like for example, I remember this
scenario in Microsoft where

customers were having like a bunch
of like SQL Server Active-Active

across regions and then this was
a bank and then 1 of the region

went down And then every second
is like thousands of dollars.

And then they immediately pivoted
to the other region and it

kind of worked seamlessly.

Nikolay: But doesn't it mean that
all clusters need to perform

the same writes?

And I didn't buy this idea that
replaying changes through logical

replication is less expensive than
applying them initially, which

was advertised in the BDR documentation.

I didn't buy this.

I hope to find time to test it
properly and write a blog post about that.

So far didn't find time yet for
this.

It's an interesting topic because
like in BDR, it means like, for

example, you have multiple regions,
4 or 8.

I know such cases as well.

And they suffer because everyone
needs to perform the same writes.

And if there is a spike in 1 region
or this, it's not a resilient

system at all.

I don't understand this.

Sai: Yeah.

A hundred percent.

I agree.

I think, I mean, that's the reason
there is a big opportunity

there.

And I recently saw one company,

like what the PGYedge who are like
trying to do something like

this.

Nikolay: Yeah, new wave of this.

I know every CTO is dreaming about
this, I know it.

Because we are like, we work in
multiple regions.

Maybe we should first check that
all our physical standbys are

in different availability zones
than primaries, right?

Sai: There are more basic things.

Nikolay: Yeah, but multiple regions
is a great thing to have,

but still I'm very skeptical.

But I see in DBA's mind, there
is a mind shift as well.

Like 10 years ago, all DBAs said
you don't need this.

But things are changing.

So in Postgres 16, this to avoid
loops, infinite loops, right?

This feature, it's interesting.

So I don't understand use cases
and how we can scale writes if

we don't split them, like in sharding.

We split them in sharding, and
that's great.

Here, everyone needs everything.

Well, I'm very skeptical.

I don't understand this part of
I don't understand this part

of landscape.

Sai: Yeah, 100%.

And like, I think it becomes very
critical in tier 0 use cases,

guys.

I mean, not like, I mean, tier 1,
tier 2, tier 3, I think it's

kind of lesser, but like, it's
more these tier 0 where like,

it's like a Chase bank or something
like that right? Like when

it becomes like hyper.

Nikolay: But you think it's possible
to build good system?

Sai: No, I don't think it's, I mean
with Postgres I am also very

skeptical, but I think there is
an opportunity there, right?

Like, and community, both community
and like, you know, I mean,

I mean, community will be very
critical here, right?

Like, I don't think that it can
happen just by logical replication.

Nikolay: Maybe if we build system,
categorizing data in tiers

as well and replicating writes
only for most critical data between

regions, right?

Michael: We're probably tiptoeing
into a different topic here.

It's

Nikolay: not different.

It's not different.

Many people think logical leads
to multi-master, definitely,

like so-called old term.

Michael: But I think the tier 0
use case feels like it's a long

way from being supported by native
Postgres.

There are a lot of providers out
there for that kind of thing.

And I think the general term for
it is being called distributed

SQL or like that seems like dist
SQL is what I've been referring

to.

But I would warn people against
using logical for this in the

short, like anytime soon.

Yeah.

Nikolay: Okay.

Sai: I agree.

Nikolay: Okay.

Then what, what other questions
do you have?

Michael: I didn't have anything
else.

I wanted to thank Sai.

Did you have anything else Nikolay?

Nikolay: Well no.

Thank you so much.

It was interesting.

Thank you for coming.

Sai: Absolutely, guys.

I really enjoyed chatting with
you and thanks for inviting me.

Nikolay: I hope you won't stop
posting interesting technical

posts.

Sai: No, no, no.

So content is our currency, guys.

I mean, the thing is, for me, I'm
building the company because

I'm very curious, right?

Like now the thing that is like
haunting me is that slot

invalidation, not like, I'm not understanding
why it can happen.

Right.

Like, so, so I've been like, so
because of curiosity, we will

be publishing a lot of content,
but

Nikolay: yeah, benchmarks, graphs,
data and so on.

Reproducible also.

Yeah, that's great.

Thank you so much.

Michael: And also, if any of our
listeners have seen that as

well, have any theories or have
a reproduction case for it, let

us know.

Nikolay: Mm-hmm.

Absolutely.

Michael: Good.

Wonderful.

Thank you.

Thank you both.

Take care.

Sai: Thanks, guys.

Bye-bye.

Bye-bye.