Michael: Hello and welcome to Postgres.FM, a weekly show about

all things PostgreSQL.

I am Michael, founder of pgMustard, and this is Nik, founder

of Postgres.AI.

Hey Nik, how's it going?

Nikolay: Good, good.

How are you?

Michael: Yeah, keeping well, thank you.

Nikolay: Yeah, so a long time, obviously.

We skipped last week because of me doing some stuff at work and

with family, so couldn't make it.

Thank you for understanding and I'm glad we are back.

Michael: Yeah, and no complaints from listeners, so thank you

everybody for your patience.

Nikolay: Nobody noticed.

They are still mostly catching up, we know.

Yeah.

Yeah.

Somebody shared on LinkedIn that like, oh, I discovered PostgresFM

and while I was walking the dog, listen to all the episodes.

It took me like almost 2 weeks.

Like, okay, once 1, 164 episodes during 10 days it's challenging

right it's like 16 episodes per day

Michael: yeah that's binge watching

Nikolay: Yeah It's you need maybe 2x or 3x speed and yeah, it's

insane.

So This topic we have today.

It's not my originally.

It's from Maxim Boguk.

I Probably mentioned him a couple of times.

He always has very interesting ideas and approaches to solving

hard problems.

And, I remember he said, he told me it was maybe half a year

ago, maybe more.

He said, I squeezed more than 10 terabytes per

hour when copying data directly from 1 machine to another.

And for me, the standard was 5 years ago, it was 1 terabyte per

hour.

I think I mentioned it a few times.

We had an episode about how to copy Postgres from 1 machine to

another.

And 1 terabyte per hour, it's like this.

Okay.

Modern hardware, we know there's a throughput for disks, throughput

for network, throughput to disks on the source and on the destination,

both matter here, parallelization, and SSD is good with parallelization.

So basically, golden standard for me was 1 terabyte per hour.

And if you need to create WAL-G or pgBackRest backup, it was 1

terabyte per hour at least.

And then we saw like 2, 3 terabytes if you raise parallelization

with modern hardware.

But it was still about not local
NVMe, but about EBS volumes

or PD SSD disks on Google Cloud,
persistent disks on Google Cloud.

And so like traditional disks.

By the way, those are also improved.

EBS volumes are pretty fast these
days and the Google Cloud also

has hyperdisks which are impressive,
but not as impressive as

local NVMe disks, right?

Maxim told me that, of course,
he was using local NVMes.

It was some self-managed Postgres
setup with special machines,

so local disks and no snapshots
like in cloud and so on, and

backups to where?

To different machine or to S3?

It doesn't matter here.

The idea was how fast we can provision
a replica, because sometimes

we need it, for example, 1 replica
is down or we reconfigure

something, we need to build a replica.

And normally if we have cloud disk
snapshots, these days we use

cloud disk snapshots.

By the way, there is great news.

Andrey Borodin implemented in WAL-G,
Andrey is a maintainer

of WAL-G, implemented native
support for CloudDisk snapshots

in WAL-G as an alternative to
backup push, backup fetch.

So instead of full backup or delta
backups, you can rely on snapshots,

AWS, GCP, or others provide.

Michael: Yeah, I saw this.

Is it a draft PR at the moment?

Or what's the status of this?

Nikolay: It was just created a
few days ago, early.

It was vibe-coded mostly.

Andrey is a great hacker, but I
think our hacking Postgres sessions

contributed to this idea.

Let's do more code writing with
LLMs.

But of course, the review will
be thorough.

I think it's already happening.

I saw many comments and the result
will be improved and so on.

So no hallucinations will be allowed.

Right.

So all covered with tests and so
on.

Anyway, there is this idea that
instead of full backups, we can

have snapshots, but at the same,
like I see like 2 big trends.

1 trend is, okay, if we have snapshots
for disks, let's use them

more and more and more and rely
on cloud capabilities, depending

on cloud.

Of course, such backups need testing
and so on.

In parallel, there is an old idea,
we should not be afraid of

using local NVMes.

We know they lose data if the machine
is restarted.

Ephemeral disks, right?

But it's okay.

These days, if a machine is having issues, we usually just re-provision

it anyway.

And switch over or failover doesn't require restart, so it's

very seamless.

It means that we can live with local NVMes, right?

And additional momentum to this idea recently was created by

PlanetScale, which came to Postgres ecosystem and started showing

very good looking pictures in terms of benchmarks and real latencies

from production systems.

Of course, you cannot beat local NVMe disks with their million

to 3,000,000 IOPS and amazing throughput, like more than 10 gigabytes

per second writes and reads.

It's insane.

Because for regular disks we have 123 gigabytes per second only

and that's it.

And IOPS, 100 to 100,000 IOPS maximum.

Michael: And I think for me, the critical part is if we have

an HA setup with failover in place, then we don't need the durability

of those disks, that we don't need there to be 6 cloud backups,

you know, so because we're gonna fail over anyway So yeah, it's

that that insight and I think they did a good job of making that

very clear If you've got an HA set up we can make use of the

NVME drives.

Nikolay: Yeah, by the way, I recognize I'm mixing throughput

and latency.

Anyway, local NVMes are good in both aspects of performance and

like almost or sometimes full order of magnitude better than

network attached disks and so on.

But you don't have snapshots.

So 2 big downsides, you don't have snapshots, and there is a

hard stop in terms of disk size.

If you plan to grow 200 terabytes, you definitely need something

like sharding, or anyway, some way to split.

Only if you have it, if you master it, then in this case, local

disks are great for, for like long-term planning, but for long-term

planning, Everyone should remember also that EBS volumes and

PD SSD or something, hyper disks are limited to 64 terabytes

and both CloudSQL and RDS, this is hard limit for them as well,

64 terabytes.

This is cliff or wall, very hard 1.

And Aurora has 128 terabytes, double of capacity.

But anyway, if you self-manage, you can use LVM, of course, to

combine multiple disks, and I think some companies already do

it.

But in the case of local disks, it's really hard stopping, nothing

to combine, unfortunately.

Anyway, you usually combine multiple disks when you have terabytes

of data.

In my benchmarks, in our benchmarks, we combine multiple disks

on the i4i instances.

So there is a hard stop in terms of capacity.

I think on i4i it's 40 terabytes or something like this.

It's impressive, right?

So if you think you won't grow to that size very soon, it's a

very great alternative to consider if you go self-managed or

some self-Kubernetes managed setup compared to RDS.

Because it gives you an order of magnitude better performance,

right?

Michael: Yeah.

By the way, when you say our benchmarks, do you mean in the blog

post you did with Maxim?

Nikolay: Yeah, yeah.

So the original idea was by Maxim.

He did the majority of initial work, created the recipe.

I will tell the recipe, like why it existed.

But the idea was we need to copy from 1 machine to another as

fast as possible.

We don't have snapshots because it's local disks.

Either fully like maybe your own data center or it's these instances

with local disks like i4i, i7i.

I remember I explored this idea very long ago, 10 years or so

maybe, yeah, 10 years maybe ago with i3 instances at that time.

I really liked the cost of the final solution because the disks

are included, instance costs are slightly higher, and performance

is great.

So we did a lot of benchmarks using i3 instances, even spot instances,

I remember it was great.

So you get very cheap, very powerful machine and do with Postgres

many things.

So anyway, the goal was how to clone to provision a replica,

for example, or to provision a clone for experiments.

Michael: Yeah, I wanted to ask about this because in the benchmark,

for example, you turned off checksums.

And it was interesting to me, Because if you're provisioning

a replica, presumably you wouldn't do that.

But if you were doing it for some other reason, maybe you would.

Yeah, so what are the use cases here where this would make sense?

Nikolay: Well, if it's a clone for experiments, we don't need

to be super strict.

And this option, I think, exists mostly for backups to make them

super reliable and so on.

Of course, and also, you know what, this is an extra feature

that BackRest has, which it's actually a luxury to have it.

So yeah, of course, it would be good to see experiment with checksums

enabled of course results should go down but for some cases it's

a it's appropriate to disable that that check

Michael: that's a good point actually so if we don't have it

it's not Is there a risk of introducing corruption, or is it

more that we persist...

Like, if our primary is corrupted,

Nikolay: is

Michael: it both?

Nikolay: Let me ask you this, if
you copy data using pg_basebackup,

Who will check this?

Who will do that?

Michael: Yeah, good point.

Nikolay: Or you use rsync.

Traditional ways are rsync and
pg_basebackup.

This is the most traditional way.

Both are single threaded.

That's the point.

They are single threaded, so, and
there is no checksum.

This is an extra feature.

It's great that DBLab implemented
it.

And it would be good to compare
how it affects this amazing performance.

But anyway, the original goal was
to be able to copy data with

traditional way, not involving
backups this time.

Because usually, we basically,
like normally, If not snapshots,

we just fetch from backups using
WAL-G or pgBackRest.

This is also a very fast way, and
there you can control also

parallelization, you can provision
quite fast.

And S3 and GCS on Google Cloud,
they are good with parallelization,

like 16 threads, 32 threads.

You can speed up significantly,
you can increase throughput of

backing up or restoring from backup.

But in this case, OK, we don't
have those backups.

We just want to clone from 1 machine
to another, that's it.

And the problem is, pg_basebackup
is still single-threaded.

There are discussions and even
patch proposed to make it multi-threaded,

but apparently it's not trivial.

And I don't see current work in
progress, So I think it will

stay single-threaded.

So Maxim came up with the idea
that we could use pgBackRest node

to create backups in S3 and then
restore, because it's basically

copying twice.

If you have them already, good,
just restore.

But if you don't have them somehow
or cannot involve them somehow,

you just have 2 machines you need
to clone.

In this case, he just came up with
idea, let's use pgBackRest to

copy from 1 machine to another,
that's it.

It's not its primary job, right?

But why not?

And he told me, like I said, months
ago that he achieved more

than 10 terabytes per hour.

Yeah.

Which is great, which is like absolutely
great and impressive.

Michael: And multiple times what
you thought was, you know, considered

good.

Nikolay: Yeah, so I already, from
this fact and others, I already

recognize that Like 1 hour, 1 terabyte
per hour is not a golden

standard anymore.

It's outdated.

We need to raise the bar.

Definitely.

So maybe 2, at least, I don't know,
maybe more 3, 4, how much

like should be okay for modern
hardware for large databases these

days, maybe 5 terabytes per hour
should be considered as like

good because you, you see, you
build some infrastructure and

you see some numbers and we have
clients, not 1 client, many

clients who come to us and complain,
okay, We will backups take

like this amount of time, like
10 hours.

What's your size of database?

200 gigabytes, 10 hours, 200 gigabytes,
something is absolutely

wrong, you know, maybe like, let's
see where situation is, where

is the bottleneck.

And it can be disks, in many cases,
but maybe network, maybe

something else, mostly disks.

But anyway, this is not normal.

Sometimes it's software and lack
of parallelization and so on

but it's not okay to live with
these bad numbers these days,

right?

It's already...

We have very powerful hardware
usually.

Michael: Just to ask the stupid
question, what's the negative

impact of it taking 10 hours?

Nikolay: Well, it affects many
things, for example, RPO, RTO,

right?

You create backups, you recover
a long time, it affects RTO.

How much time, recovery target
objective, how much time do you

spend to recover from disaster?

You lost all the nodes, how much
time to get at least something

working, at least 1 node working.

If it's 10 hours for 200 gigabytes,
it's a very bad number.

You need to change things to have
better infrastructure and software

and so on.

And This is 1 thing.

Another thing is when you upgrade,
for example, sometimes you

need to clone, our recipe involves
cloning, a recipe for 0 downtime

upgrades.

And if it takes so many hours,
it's hard to think what will happen

when you will have 10 terabytes?

How many days will you need, right?

So it's not okay.

This requires optimization earlier.

Michael: And also for provisioning
replicas, a certain amount

of time taken is always going to
be acceptable, I guess, because

it's not the primary, right?

Like we've maybe

Nikolay: lost...

It depends.

If it's only primary you are out
of replicas is dangerous even

with the most it's I would be running
with 2 nodes I would spend

as much as less time as possible.

Michael: So when you say 2 nodes,
do you mean with 2 replicas?

Oh no, okay, so 1.

Nikolay: Primary and 1 standby,
it's already degraded state.

We need a third node, right?

To be in, with 3 nodes, you have
how many nines, 12 nines or

so?

If you check the availability of
EC2 instances and just think

about 3 nodes and what's the chance
to be out of anything, this

will be I think 12 nines or something,
as I remember.

Michael: That's a lot of nines.

Nikolay: Yeah, yeah, and that's
great.

So it means like almost 0 chances
that you will be down unless

some stupid bug which propagates
to all the nodes and puts Postgres

on knees on all nodes simultaneously.

Anyway, if we integrate it, and
especially if it's just 1 primary,

it's a very dangerous state.

Like, I like, we have cases in
some clients who run only 1 node

and like, we discuss it with them,
it's not okay, but somehow

these days it's, they just survive
because the cloud became better,

you know, like it doesn't die so
often.

Nodes don't die so often as before.

Michael: I also think there's this
phenomenon where if you're

in US East 1 and you go down there's
so many other services that

are down at the same time that
you kind of get not a free pass

but like they get away with it
a little bit more.

Nikolay: You're in the club right?

Michael: No no no

Nikolay: you are down others are
down like we are we are in the

same club.

Michael: Yeah, I'm not in that
club, but yes, I think a little

bit.

Nikolay: No, I think, yeah, we
have also customers who got in

that trouble as well, and they
seriously think it's not okay,

and they need more region set up,
And I think this drives right

now improvements in many companies
and infrastructure and so

on.

Michael: I'm glad to hear that.

Nikolay: Yeah.

Anyway, and with our 1 node is
dangerous, like I would provision

more nodes sooner.

And in some cases you cannot leave
with 1 node.

If you already relied on distributing
read-only traffic between

among multiple nodes, 1 node won't
be capable of serving this

traffic.

Michael: Yeah, so we've established
speed matters.

Of course.

Sorry, time matters, therefore
speed matters.

How do we go faster?

Nikolay: Yeah, so pg_basebackup,
I expected like, okay, it's single

threaded, so I expect something
like 234 hundred megabytes per

second maximum.

And this is what also my expectations
fully matched with what

I saw Maxim shared with me.

But when I started testing I took
2 nodes i4i, it's not the latest

it's a third generation Intel scalable
Xeon which is quite outdated

they have already fifth generation
i7i so 2 nodes 32 cores it's

128 vcpus more than terabyte of
memory I think both, and 75 gigabit

per second network.

And the disks are like, I think
it's 3 million IOPS I remember,

I don't remember how many megs
per second, gigabytes per second.

It's definitely somewhere like
10 or more gigabytes per second

disk throughput maximum.

I think 8 disks already in RAID.

I took Ubuntu.

I installed Postgres 18 and this
was my mistake.

Because this gave me very good
speed for pg_basebackup.

Unexpectedly.

I saw gigabyte per second.

And I was like, oh, what am I doing
wrong here?

Because it's too fast.

And then I realized it's Postgres
18 improvements, right?

io_uring.

Michael: Well, yeah, I saw you
said that in the blog post.

But did you run it deliberately
with io_uring on?

Nikolay: It's by default.

Michael: Well, no, it's not io_uring
by default.

But it is using something similar
it's using something similar

it's like

Nikolay: that's some pre-fetching
or something but it's definitely

nothing

Michael: yeah well it's it does
have 3 work 2 or 3 work processes

by default, and it is an asynchronous
I/O thing.

Nikolay: No, I didn't change that
setting.

So this is probably an accuracy
in my blog post.

Michael: But maybe not bad, because
it might still be AIO.

It might still be asynchronous
I/O, just not io_uring.

Nikolay: So, right, there is setting
I/O method, right?

And default is worker, right?

Yeah.

And it's asynchronous I/O using
worker process.

Yeah, it's not io_uring, I need
to make correction.

Michael: But this is actually interesting,
because I think the

default is only 2 or 3 workers,
which means if you increase that

number, you might see triple, like,

Nikolay: 3 workers, you

Michael: got triple.

Yeah.

So you got roughly triple the

Nikolay: exact in my expectations
for 300 yeah

Michael: but if you increase that
number if you increase the

number of workers you might be
able to get even more

Nikolay: doesn't mean I need to
spend a few hundred dollars again

for these machines?

I guess so.

Michael: Or an exercise to the
reader.

Nikolay: How it worked.

I started them at 6pm or something.

I worked with them.

Cursor did a lot of work.

Like I just explained control that
I connected T-Max, Iostat,

IOTOP, everything.

I see how many threads, everything,
like htop, many things.

So I see that it's doing work as
I do.

Many iterations to Polish approach.

And then I realized, okay, it's
already 9pm, 10pm, but I cannot

drop it because it's already provisioned
and it took time to

create 1 terabyte database.

I had comments from Maxim as well
at 1 terabyte is not enough.

Like, okay, I agree.

I should do 10 terabytes.

So I guess I need to redo this.

I have homework to do.

Michael: You don't need to redo
it.

It's just an interesting-

Nikolay: It's interesting to me
as well.

Because I also want to see Postgres
17, you know?

Yes.

And different settings here.

I think it's interesting how pgBaseBackup
can behave with various

settings.

More workers, io_uring as well,
sync as well, right?

Synchronously.

So not asynchronously like here,
but synchronously.

It should go down.

The throughput should go down.

Should go

Michael: back down.

Nikolay: Yeah, yeah, yeah.

So now, obviously, you just make
me do another round of this

experiment right

Michael: well sorry but I also
think like this someone else could

do this right someone else could
do this

Nikolay: yeah I I will have it
in my to-do list, but if someone

who is listening to us is ready
to repeat, maybe they have already

some good machines.

Unfortunately, I don't have credits
anymore.

Like I did in my company, had a
lot of credits, but not now.

Maybe someone has credits and can
provision big machines and

just avoid extra spending Because
I think a few more times it

will be already $1,000 just to
check this setting.

And I'm not interested in checking
on smaller machines and then

like extrapolating.

No, no, no.

It's boring and not serious.

So it should be big machines.

I would also check i7i Because
they have 100 megabit per second.

It's versus 75.

So also extra I Think we can exceed
10 gigabytes per second And

so what I've got with pgBackRest
with pg_basebackup 1 gigabyte

I think you are right in terms
of 3 workers.

This is why, right?

Interesting to check more workers.

With pgBackRest, I increased
parallelization and my blog post

has this like the Graph showing
how throughput was growing with

more and more workers and then
situation happened on that network

So I achieved 36 terabytes per
hour And that was also exceeding

my expectations.

I was hoping like be close to 20
maybe, right?

But this machine's,

Michael: yeah.

It sounds fake to me.

It just sounds like not believable.

But it obviously is impressive.

Nikolay: Yeah, so i7i with 100
megabit per second should give

maybe more than 40, right?

Maybe approaching 50 TB per hour.

This raises the bars and expectations
what we should have in

our systems these days.

So it should be normal, 5 terabytes
per hour should be normal

now, I think.

Answering my own question some
minutes ago, right?

10 terabytes should be not surprising
already, if you have local

disks.

With BS volumes, different.

Michael: Yeah, for me, I guess
the surprising thing is that we

can saturate the network.

Like, it then becomes about your
network, right?

I think.

And the number of cores.

Nikolay: Yeah.

Yeah, if you saturate network,
the idea is of course, let's try

compression.

So I did.

And compression improves things,
but, so it shifts the saturation

point, but it's like quickly saturated
again, and it was not

helping to achieve more.

Maybe I saturated the disks there
actually.

So if you look at the picture...

Michael: I'll pull up your blog
post as well.

Yeah, because you hit a peak around
32 parallel processes.

Nikolay: Ah, also SSH overhead.

I've got good comments on Twitter
that it would be good to check

TLS.

It would require effort to configure,
but it should also reduce

some overhead and throughput should
be improved.

Wow.

So I guess 36 terabytes per hour
is not the current limit for

modern hardware in AWS.

We can squeeze more, right?

So we can probably squeeze up to
50 terabytes per hour.

There are several good ideas, and
also squeeze from pg_basebackup.

So this is a competition, right?

So my intention was to show like
pg_basebackup is bad, single-threaded,

forget about it, here's the recipe.

Now with Postgres 18, it's not
that bad.

It can be tuned additionally, as
you say, probably, right?

And we can have very good speed
with it as well.

So it's interesting.

And 1 gigabit per second is more
than 3 terabytes per hour.

Michael: Yeah, wow.

Nikolay: Yeah, 3, 600 gigabyte
per hour, right?

It's more than 3 terabytes.

So it's already like good enough.

And these are default settings.

So the answer, should you use pgBackRest
Recipe?

Well, it depends.

With Postgres 18, with older Postgres,
I definitely think you

will be limited by 3, 400 megs
per second.

That's it.

Because of single-threaded nature
of the base backup.

And I like this a lot compared
to the tricks we did in the past

with rsync.

rsync is also single threaded and
I really don't like that.

Why is it so?

It should be implemented.

People use GNU parallel and so
on, but it so feels cumbersome,

right?

So it's not a lightweight approach
to write some things and then

control the threads and so on.

With rsync, it's definitely single-threaded.

pg_basebackup in Postgres 18 basically
shines compared to rsync.

And also it orchestrates all the
stuff additionally like like

connecting to the primary doing
like telling it I'm copying you

right pg_start_backup, pg_stop_backup
basically right so you

don't need to remember all those
things and it's also standard

officially.

Michael: Although I would say I
was looking because of researching

for this I was looking at the
pgBackRest docs and GitHub repo

just earlier today and it's incredibly
well maintained like I

was surprised to see

Nikolay: yeah we're moving to David
Steele, yeah it's great

Michael: yeah kudos to David but
also just in terms of viewing

a project for the first time in
a while I hadn't seen I hadn't

like looked into it in detail for
a while.

And I think I was just a bit surprised
when I looked at an open

issue.

I was like, there's only 54 open
issues?

Like that's, for the size of the
project, for the age of the

project, That for me seems like,
and by the way, I'm not taking

that number on its own, looking
at what they were, like some

of them are like feature ideas
from a few years ago that are

just left open because they're
still good ideas.

Some of them are new things.

Nikolay: That's good because there's
approach when people just

close if it's like, okay, no comments
for a month, let's close.

And I absolutely hate it because
if you reported a bug which

is not fixed, CloudNativePG, hello,
like they just closed because,

right?

Because it's a little bit of a
discussion, but bug is still there

and everyone around agrees it's
still there.

Why do you close my issue?

It's a bug report.

So next time I won't write anything,
right?

And I told it everywhere.

And in CloudNativePG, I'm not
going to contribute anyhow else.

So this is like, let's just keep
issue set lean, right?

It's not okay.

If it's a serious problem of idea,
it should be there.

So the fact that there are all
the issues there is great.

Michael: Yeah, yes.

So all I meant is, you kind of
get these, Sometimes you get red

flags when you look at a new project
or like if you if you see

a project And it's got 20,000
open issues for me That is a red

flag like not that it's that there's
lots of issues But that

no one's maintained no one's closing
the ones that are duplicates

or things.

So it was more just like, I wasn't
getting red flags and then

there were loads of green flags
looking at how like recent PRs

and it just, it seemed like a route.

So I know it's not in core, but
it does still feel like a very

tried and tested product that's
really well maintained.

And I know a lot of people are
using it.

So it feels pretty much as close
to core as like a...

Nikolay: What if you see hundreds
or thousands of issues open,

but many of them are recent and
there is activity.

Not well maintained or a mess.

Michael: Spam issue?

Like how can you create hundreds of thousands of issues?

I just haven't seen a repo like that so maybe I have to like

reassess but yeah it's yeah.

Have you seen a repo like that?

Nikolay: I have such projects.

Oh, okay.

So it's just, yeah, there are some old issues that should be

closed, you know, yeah.

Yes.

Well, If you check, well, so I agree this is a great example

of well-maintained open source project.

Absolutely great.

I agree.

I think if you check some internal repositories in companies,

sometimes it's a mess as well.

Michael: And sometimes it's beautiful.

I think it depends a lot on the maintainer

Nikolay: what can you tell about Postgres itself yeah

Michael: well we don't have an issue track

Nikolay: do yeah no issues no no issues right well like

Michael: we should get that would be a good t-shirt.

No issues.

No issues All right, let's we get back to the topic

Nikolay: back to the topic so David Steele reached out to me commenting

and it's great So some inaccuracies in my blog post to fix.

And also, obviously, it inspired him to, as I understand this

idea was already for quite some time to have the feature to basically

issue renice, to reprioritize some processes, right?

So now there is a pull request to be able to change priority

of pgBackRest processes, which is great.

Michael: Yeah, so this is super cool.

So this is the idea that because you're running this on production,

you might not want to, you know, let's say you're quite high

CPU on your primary.

Nikolay: Compression.

Yeah.

You use compression, so you need CPU for pgBackRest, for example.

Michael: Yeah.

But you don't want to affect your current traffic, right?

It reminds

Nikolay: me of old days, like 2003 or 2004, when we also...

It's so strange, like, it was...

We couldn't think about terabytes those days, 20 plus years ago.

It was too much.

Terabyte is a mind-blowing number.

But we used, I remember, renice, it is renice, ionice, something,

I barely remember from those.

Michael: Just NICE, yeah?

Nikolay: Yeah, NICE and so on.

It's strange that this word is used to change priority, right?

I don't know.

Do you know details here?

Yeah, no, I don't.

Yeah.

Yeah.

So I remember from those days we used it, but also I remember

I was struggling to see good effects.

So now if this pops up again, I would check, test it properly

again, how exactly it helps, you know, but this is in to do.

Michael: And really cool that your, that these benchmarks helped

prioritize that work.

So cool.

Nikolay: Inspires to do more benchmarks as well.

Michael: Always.

I get stuck in that loop sometimes, not just with benchmarks,

but you know, you think you've used to implement a feature, then

you think of a better way to implement it, then you think of

a better way, and you think of a better way, and you get stuck

in that loop of constant, yeah, or blog posts.

Nikolay: Yeah.

This post is definitely not the final.

So somebody should achieve 50 terabytes per hour.

This is a challenge.

Yeah, I like it.

Yeah, Who can do it?

Who will be the first in the Postgres community reporting 50TB

per hour?

Michael: Yeah, let us know.

Although, I did see there was an issue that David linked us to,

where somebody maxed out a 100-core machine, somebody maxed out

a 100 core machine and it convinced him to increase the, he had

a process max at 99 before then.

So I thought that was quite funny that, you know, there was a,

the limiting factor was this hard-coded yeah limit

Nikolay: number of maximum number of vcpus on AWS already I think

has it exceeded already of 1000 or no I remember 700 vcpus with

also fifth generation into 0 scalable so it's already like just,

we have a lot.

Michael: Bear in mind your test maxed out at 32 processes, right?

Right, but this 1...

Nikolay: Bumped into network.

Michael: Yes, okay.

Nikolay: And then when I increased compression it shifted, right?

So it shifted to 64 cores.

Michael: So I'm wondering what do you think the person in the

open issue back in 2019?

Maxed out their hundred core like what do you reckon their?

Throughput was

Nikolay: I Lower it's interesting, but but maybe you know like

we need compression not only to battle network throughput limit,

but we also, if it's actual backup,
we want, sometimes we want

it to take as much, as less space
as possible in S3.

Right.

Just to pay less, maybe.

Right.

In this case, we can have aggressive
compression and CPU consumption

high.

In this case, it can be 128 cores,
for example, third generation

of Intel Scalable Xeon has 128
cores maximum, I think, at least

in clouds.

It's N2 on GCP and I4I, as I used
on AWS.

But I wonder a lot, how come AWS
created 7 or 800 VCP use machines?

It's like, wow.

So like 10 plus terabytes of memory.

Wow.

Yeah.

Michael: Yeah.

By the way, when you keep saying
there was the Google disks earlier,

I think you were saying that every
time you said that, all I

could hear was PTSD.

PTSD.

Nikolay: PD, SSD.

System disk, SSD.

Yeah, yeah.

I

Michael: heard PTSD.

And then when you're saying eye
for eye, all I can think of is

the saying, an eye for an eye in
the world is blind.

So now...

Nikolay: Eye for eye.

Michael: Yeah.

Nikolay: Oh, by the way, 1 interesting
thing, like EBS volumes

these days also are built on top
of, NVMes.

I don't know how's it called, Nitro
architectural.

It's always the new name.

I don't follow the, all the terms,
but I remember 2 gigabytes

per second and even more.

Right.

So I think we can squeeze a 5 to
6 7 terabytes per hour on modern

EBS volumes as well

Michael: even on EBS Wow

Nikolay: yes yes yes so so forget
about 1 terabyte per hour

Michael: it's old you can do better

Nikolay: yeah definitely better
if you have a large database,

do better.

Michael: Nice.

All right.

Good.

Thanks so much, Nikolay.

I enjoyed that.

Take care.

Nikolay: Bye bye.