Michael: Hello and welcome to Postgres.FM,
a weekly show about all things PostgreSQL.

I am Michael, founder of pgMustard.

I'm joined as always by
Nik, founder of Postgres.ai.

Hey, Nik

Nik: Hi, Michael

Michael: And we have two
special guests with us today.

We have Qian, who is CEO and
co-founder of DBOS, and Peter,

CTO and co-founder of DBOS.

Welcome both

Qian: Thank you

Michael: I wonder if one of you could
give us a bit of the backstory of DBOS.

I think it's quite a cool story

Qian: Yeah.

So we started DBOS as a research project.

I think it was started quite in the middle
of pandemic in summer 2020, when Professor

Mike Stonebraker heard a talk from Matei
Zaharia, the profes- our advisors and

also co-founder of Databricks, where they
were talking about how Databricks were u-

was using Postgres to do some scheduling
work because, distributed scheduling

and cluster management was a mess.

And they were talking about how to
use a database to simplify that.

And then I think during that summer we
talked and discussed, like, how we can

u- extend that idea to make database
more helpful to build, reliable systems

to even replace the operating system.

So that's how we started the original
research project called DBOS.

And then Peter and I were the co-leads PhD
students working on that research project.

We published several
papers, did a bunch of work.

And then what we found was that, the most
interesting thing was actually to use the

database as a foundation for distributed
systems, to build distributed systems.

And that's why when we both graduated
in 2023 we decided to do, to

commercialize this idea, to do a startup.

And now we are working on
DBOS Inc., the company.

Nik: that's cool And I must admit
I observed queues in Postgres

for so long, like 20 plus years.

And somehow when DBOS DBOS was
announced, of course it it attracted

my attention, but I didn't quite get
it like Because like w-w-workflow,

like what i-what is it, right?

But then I started to realize,
actually, to realize that in any

business there are workflows, and to
keep the state and all the history and

so on in database, it's very natural.

So this is a very good idea.

And I think it's, it's-- it revives
the idea that queue in that database

is not only possible, but it's
possible in quite efficient way.

But you need to… With Postgres,
you need to be very careful, right?

Because of some, some things that
might happen if you implement

queues and workflows in naive way.

But I don't quite get it
still why operational system.

I understand when Andrei Karpaty
talks about like AI as a CPU in

the center, like memory, like every
p-peripheral devices, everything.

I understand this analogy, but here
I don't quite get the analogy why

is there is some intent or idea to
implement whole works operational

system in, in da-database.

Like what's your view?

Peter: Yeah.

So what we're doing as a company is
using the database as a foundation

for distributed workflows, and then
for things around the distributed

workflows like queues, like messaging,
like notifications, and so on.

I think the core idea here is that
as your program runs, you regularly

checkpoint the state of that program
in a Postgres database so that if the

program ever fails, if anything happens
to it, if it crashes, if it's interrupted,

it can reload those checkpoints to
recover from exactly where it left off.

And that, durable workflows idea backed
by Postgres is the core idea of DBOS,

using Postgres as an orchestrator to
make durable workflows that run really

w- that run with high performance
and that are really simple because

all they have is, all they have,
all they rely on is your database

Nik: Yeah, this makes sense.

I understand that.

I'm just still processing the idea.

It's like whole operational system.

Qian: I think our goal is that when
we think about operating system,

we should not be limited to the
operating system we're running

our, on our laptop or desktop.

Thinking like anything operational,
I think Kubernetes can be called

as a distributed operating system.

Thinking about like anything that
can run your application as like a

meta name for like operating system.

So I think nowadays, especially with
AI, operating system is like cool again.

Everyone is calling themselves an AI
operating system or, like something

AI-powered operating system for
engineers, for sales, for whatsoever.

So I think operating system now, like
the idea is more about how to properly

run your applications in any environment,
like in a distributed environment,

in a highly available environment.

It's about how to provide the
guarantee and provide kind of an

illusion that you are ru- like normal,
like modern developers no longer

directly write or interact with
low-level operating system primitives.

They usually interact with some- something
like Kubernetes or cloud resources.

So like you can think of the cloud
as an entire operating system.

So like I think the name like DBOS right
now really focuses on how to operate

those, applications at scale and reliably.

So it's I would say it's eventually
at a grand vision, we hope everything

will be backed by a database,

Nik: Yeah.

Qian: then that will be like
a super operating system.

Yeah.

Nik: Yeah this makes sense in, in
broader meaning operating system.

Yeah.

Not just like Linux or Windows

or something, Right.

I get this.

And also like there are even broader
systems like there, there are companies

like which are building like an
operational-- operating system for

businesses in broader meaning, right?

I also remember when Anthropic released
I think Ultracode with workflows.

I remember on Hacker News
top both topics were present.

One was about their workflows,
another was about some blog

post from your blog, actually.

I remember.

It was like-- And for me, it was
like this is two different things.

And I remember internal discussion in
our team actually, that we use these

workflows in Claude Code, but workflows
which are like saving some state

in database is something different.

And then I remember an aha moment when
we realized we actually have agentic

workflows ins-internally already
working, and we save state in Postgres.

So I realized it's very close topics.

I think you also use this resonating
moment when they released it and you like

to discuss workflows in database, right?

So I have one specific technical question
though because Peter, you mentioned

that if something crashes, right?

And ch- this checkpoint and database were
all good, like we ha- we have stayed safe.

I don't like fully understand why
checkpoint is needed because we always

can replay more after checkpoint.

There are WALs and so on, right?

But anyway, we

Michael: checkpoint

Nik: Ah, okay, a checkpoint there.

But what wonders me, if, for example,
we se-send an email but we don't

know the result of that yet, right?

I'm like moving us slowly to the topic
of two-phase commit or something, right?

How do you handle that?

Peter: Yeah.

So if your step is taking an action
in an external system and it fails

in the middle of the step, then we're
going to have to retry that step, and

that, that's unavoidable in any kind
of workflow or durability system.

So our recommendation then is for
individual steps that take-- have

side effects, external systems
to make those steps idempotent.

And, we provide help for that.

Every workflow has a unique ID
that you can use as an idempotency

key in systems that support that.

Many email systems do
support idempotency keys.

Some problems are fundamental
to distributed systems.

If you're taking a side effect
somewhere else and it fails and you

don't know whether it succeeded,
you're gonna have to retry it.

And so you should really try to make
sure that, that action's idempotent

Michael: And another thing I didn't
fully appreciate till recently, I kept

focusing on the word durable I see
durable workflows mentioned quite often.

I'm thinking about reliability and
retries, and that all makes sense.

The thing I didn't really
appreciate was actions that

quite-- could be quite costly.

If you've got a five-step workflow and
you've spent a lot of tokens or a lot

of money, or you've sent physical goods
or something like that in steps one,

two, and three, and it fails at step
four, you can avoid the c- the cost

of redoing steps one, two, and three.

Like it, the-- that hadn't really clicked
for me, that sometimes it's about other

side effects, like might just be cost
or it might be safety or it might be

something else that you don't have to risk
re-replaying is that a big part of it at

the moment in terms of… or are there
parts of that might be helpful to explain?

Qian: Yeah, definitely.

I think especially nowadays,
everyone's building AI agents

and those new models are becom-
becoming more and more expensive.

So if you have already asked the
model about something, you want to

use that output as much as possible.

And if something crashes, you want
to resume from where it left off.

Say, like you already asked the
model what to do next, then you

should continue from that step
instead of asking the models again.

Because otherwise you have to burn all the
tokens and this will quickly become too

costly if you're having like processing
thousands of documents and then you

want to do it in a more efficient way.

That's where DBOS is used a
lot, like building agents and

building AI-powered pipelines.

I think cost saving is
definitely… It's like becoming

more and more important, I'd say.

Yeah.

Michael: Makes sense.

Interesting

Qian: And also compute resources, right?

Even if you don't have to pay those,
Even for some people who will self-host

their GPUs and - self-host their model
inference, then you don't want to

burn all the compute resources again.

It's all about efficiency.

I think yeah, durability itself
sounds like not, doesn't sound like

a very useful word, but it indicates
a lot about cost savings and also

in- indicates about after you've done
something, you don't repeat it again.

So it's indicates about
correctness as well.

Yeah

Nik: I have a question about why
should it be inside Postgres?

We know there is-- there are systems
which do the same outside, right?

Temporal is one of the
most popular systems.

I even from my personal experience,
I even had a moment when one of our

clients reimplemented our Postgres
major upgrade workflow in Temporal.

it was, It was Gadget

Michael: They spoke
about it on the episode.

Nik: Yeah, we had an episode
about that, and it was like we

always used Ansible and so on.

Of course it's very different, right?

When you have workflow, you
save everything, it's great.

But you move further, let's bring
the whole thing inside Postgres.

What are pros and cons and, why do
you think this is a winning strategy?

Peter: Yeah.

So I'd, what I'd ask is why not Postgres?

The temporal model is that you are
taking… is that you're outsourcing

the control flow of your program to
an external orchestrator, and that's

a lot of additional infrastructure.

You can either self-host it, in which case
to run your workflows, you have to spin

up highly available temporal cluster, a
database for it, which is Cassan- and its

database is Cassandra and Elasticsearch.

And now you're spinning up three
new distributed systems to run your

workflows, and all three of those systems
are totally on your critical path.

If anything happens to
them, Nothing will work.

Your workflows will not function.

Your business grinds to a halt.

So you're putting three complex
distributed systems on your critical path.

Or - you use their cloud service.

Now you're putting an external
SaaS on your critical path.

You don't need to do that.

You just need a database.

And what we're trying to do in DBOS is
totally replace all of those distributed

systems, all those external orchestrators
with your Postgres database so that you

can use your database as an orchestrator.

It's already on your critical path.

It's already something that you're
making sure is highly available.

It's already critical for
your application to work.

Why not also use it as your workflow
orchestrator so that you have

less infrastructure, so that your
operations are simpler, so that

there are fewer things that can break
and bring down everything you have

Nik: Two things here.

One is who will-- like Postgres
doesn't have scheduler, right?

You can use pg_cron or something,
or you need to still need

external help to tick, right?

To move forward.

And another thing is that if it's all
running on the primary and primary CPU,

first of all, is not infinite, right?

So you need to avoid heavy things
and have them outside still, right?

Peter: Yeah, and that's
the reason Postgres is

the orchestrator.

The actual business logic
of your workflows is still

running in your application.

The scheduler is running in your
application, for example, and

then Postgres is the orchestrator.

So your applications are running
a scheduler, and your applications

will enqueue when the sched- when
the cron fires, your, one of your

application servers will enqueue a
scheduled workflow into Postgres.

And then Postgres there
does orchestration.

It coordinates multiple servers to
make sure that only one of those

servers actually executes the scheduled
workflow when its cron tick fires.

So your applications your regular
servers are doing the work, and

Postgres is orchestrating them to
make sure that every, all the work

is being done totally reliably, that
everything happens exactly once.

Nik: Makes sense.

So this is when you say critical
path, you don't include things

that like sending email can be done
from multiple servers and they are

replaceable like stateless and so on.

Yeah, I get that.

It feels slightly awkward for
me to try to criticize the idea.

I'm a big fan of myself, like
maybe Michael should join

and find some other questions

Michael: I was actually gonna
ask a question that probably

goes even more positive.

M- or maybe not, but I would expect
there to be performance benefits to

using your already h- used database for
this rather than an external service.

So I was only gonna say What are, what are
the performance trade-offs and what do you

do to minimize the impact you're having
on primary CPU and that kind of thing?

Peter: Yeah, absolutely.

So latency in particular gets a lot better
because now the latency of executing

a step is just one write to your local
Postgres as opposed to a async dispatch

from somebody else's cloud service.

So definitely latency improves.

Now throughput then
becomes database bounded.

You can process as many
workflows per second as, as your

Postgres allows if you want.

And if you need to scale beyond
that, you need to shard across

multiple Postgres databases.

The good news is that these
operations are fairly light.

So our rule of thumb is that four Postgres
cores can handle 1,000 actions per second.

So for a large database, that means
you can vertically scale for a large

Postgres database up to… we've
benchmarked around 40,000 actions per

second on a single Postgres database.

And then of course, you need more
than that, you can shard or you can

use multiple Postgres databases.

So your throughput does become
bounded by your database, but the

good news is that your database
can handle a lot of throughput

Nik: Yeah.

Yeah.

Michael: and we're

talking… so let's say we've
got a workflow with six steps.

We've got one write at the beginning, one
write per step, and one write at the end.

So we're still only talking eight writes
total for a si- and this workflow's having

to do things in the meantime, it's not
gonna be running through them instantly.

So yeah, great.

Okay, cool.

Peter: And you can handle 40,000 of
those actions per second on one database

Nik: Yeah, I'm curious.

I researched a little bit and I noticed
y-that you started with repeatable

read isolation l-level and then reduced
it to default read committed, right?

I'm very curious what was behind that.

Is it the throughput
question or anything else?

Peter: So we use isolation levels
there's a lot of different parts

of our system that use isolation
levels for different reason.

we've designed the system so that
almost everything is a single statement.

So it can just be a, it can just
execute a recommitted, and that

makes things far more efficient.

As many things as possible execute
as a single Postgres statement,

so they can just be atomic and you
don't have to worry about isolation.

You set it to default recommitted.

The one thing that we can't do
that for is the main DQ query.

this is probably the single most
important s- query in the entire system,

and it's the query that every worker
runs periodically on every queue to

dequeue the latest work off the queue
And that's a very complex query we've

put a ton of effort in optimizing.

That query has a, actually has a
conditional isolation level, s- depending

on what, how you've configured your queue.

So if your queue uses global flow control
primitives, if your queue, for example,

has a global concurrency limit or a global
rate limit, so you say, for example,

that this queue can only execute one
operation at a time across every worker,

then that requires global coordination,
which means we have to use a high

isolation level like repeatable read.

And that has coordination costs, but
if you have those global limits, you're

probably fine paying the cost because
you're only executing one workflow

at a time across all your servers.

th- your throughput was limited
anyways in practice though, what we

see is that high throughput queues
usually only use local limits.

They might say only execute 10
workflows at a time per worker.

And those local limits don't
require global coordination.

And that means that it's perfectly safe
to run the queues query at a very low

isolation level at at read committed.

And that way you can have this query
running across hundreds of servers at the

same time, or thousands of servers at the
same time without them conflicting because

they're all running read committed.

And that lets you get this thirty, forty
thousand dequeues, dequeued workflows per

second, actions per second throughput.

And it really caps out at thirty
thousand dequeues per second on a single

Postgres across thousands of workers
because we've optimized this query

Nik: And in case of sharding
solution re-repeat-repeatable read

wouldn't be available anyway, right?

So because like global
limits it's… Yeah.

You

Peter: But in practice, global limits tend
to go with relatively low throughputs.

Nik: And it's flexible now, right?

So u-u-user can control it
and decide what, what's in.

That's great.

And I also noticed listen to
file was recently added, right?

Or not recently, maybe I'm wrong.

Qian: We've been having listen notify,
but recently we've been optimizing it

Nik: yeah.

We ha-- Yeah, I think it was last year,
Recall AI they discovered this awful

global lock on commit for Notifier, right?

What's your deal with this right now?

Peter: Listen/notify is really powerful
but has multiple performance issues.

There was that issue that… I forget.

There was that issue that reached
top of Hacker News last year.

That was actually not
the issue we ran into.

That issue was solved in Postgres 19.

That is not the issue we ran into.

that issue is that notify had
terrible perform- notify's broadcast

mode had terrible performance for
large numbers of sparse readers.

Uh,

that's actually an unrelated issue.

We do not have that.

We have dense readers.

We don't have sparse readers.

Nik: Can you explain please the
sparse readers versus dense readers?

Peter: Oh, sorry.

The problem that everybody was talking
about last year was that if you're

broadcasting to a huge number of
readers, each of which is only reading

a small fraction of all messages,
that had terrible performance.

So that's actually not
the issue we ran into.

The way DBOS is set up, pretty much
all workers are listening to all

messages and that performs fine.

The issue we ran into was
the global lock at notify.

When you commit a transaction that
performs a notify, it has to acquire a

global lock for the entire duration of
the commit, and that includes the fsync.

And what that means is that these
transactions can't be group committed.

They have to commit sequentially, which
essentially bounds the throughput of

any operations that perform notify.

You can't perform more than a
certain number of them per second

because you can't group commit.

So you're stuck waiting out the
entire duration of the fsync.

the optimization that we settled on
there to make listen notify, because

we really want to use listen notify.

It provides incredibly low
latency, and when it works, it

provides incredibly low latency.

So w- the optimization we
settled on was notify batching.

So instead of sending a notify,
originally, we were sending notifies

in a trigger every single time you,
for example, wrote to a stream.

And the optimization was instead it
was removing the trigger and instead

buffering stream writes in a background
thread and then sending all the

notifies in the batch and send that
batch e-every ten milliseconds or so.

A huge batch of notifies is
actually very fast because you

only have to hold the global lock
for the duration of that commit.

that's fast.

And what this does is this allows us to
use listen notify without re- running

it, worrying about the global lock
because we're batching commits and

the global lock's only required for
commits, and still getting the benefits

of low latency because we can still
emit these every, every few milliseconds

Qian: I think, the overall observation is
that we don't really need that global o-

ordering when we're doing the application
level streaming, like streaming,

for example, results to the client.

Then what do you-- we're leveraging the
listen/notify to do is to notify like,

"Hey, we have some results available."

So we don't really need
that strict ordering.

And I think that was the main aha moment
for us that, okay, we don't need to

limit ourselves to that exclusive locks.

We can do batching.

Yeah.

Nik: And yeah, and this is… okay.

So and also the problem is happening.

You, Peter mentioned that it was
fixed in 19, and it's happening when

commit has a lot to write, right?

So like it's this fsync.

Basically, transaction is significant in
terms of how much it's writing, right?

So th-this increases the problem.

But in your case, you have only
like brief writes in transaction.

Not a lot, right?

Not like a lot of JSONs to be
written and fsynced during commit,

So the problem shouldn't be huge
a-already if I'm not mistaken.

and you decided to group it.

This is interesting optimization.

I, I wanted to raise
this question, of course.

You use skip locked updates and
what's happening if we have some

xmin horizon blockers, so long
transactions or lagging logical slots?

Obviously, a lot of dead tuples
should be accumulated, right?

Is it a requirement if you use DBOS
at, like at high rates, at a lot

of events per second and so on?

Is it a requirement I need to watch
closely what is my xmin horizon?

Don't allow long-running transactions,
don't allow logical… lagging

logical replication slots long
transactions, so on standbys with

hot standby feedback on and so on.

Is it a requirement or you solve
this somehow I don't know, like

maybe mitigate-mitigating bloat with
repacking, reindexing and so on?

Peter: Got it.

So just to be clear, we're only using
skip locked on our own workflow status

table, and that's typically isolated
within its own Postgres database.

So our recommendation is Don't take
out ultra long-running transactions

inside the DBOS database, and you
probably weren't doing that anyways

because DBOS certainly doesn't do that.

And then the rest should be fine.

The DBOS operations are isolated here

Nik: So you don't use skip locked
for like events t- tables where

events or workflow data is stored?

Peter: It's only used for deques off of
the main workflow status table, and that's

usually pretty isolated from user data

Nik: But still, it's you need a s- single
update on every status change, right?

Peter: Yes, and that does lead
to dead tuple accumulation.

but that's a separate issue that
y-yes, at ma- at massive, at

really high throughputs, like when
we were doing these benchmarks at,

40,000 workflows per second, we
saw high CPU usage due to dead

tuple cleanup and autovacuum because
we're turning on these indexes.

And we've done a lot of work
to optimize indexes, so most,

almost every index is partial.

So most indexes are only being
maintained either for a small

subset of workflows or only for a
small subset of workflow lifetime.

But there's still a fair amount of index
churn on the order of… And this, of

course, not as hot a dead tuple, and it's
going to create a bunch of dead index.

It's going to create two or
three dead index entries that are

going to have to be cleaned up.

At really high throughputs, this can
become the bottleneck but that sets in

around 20 or 30,000 actions per second
is when we start seeing the tuple

accumulation really becoming a problem,
because there really aren't that…

Tuple accumulation become
a problem, because we have

optimized this as much as we can.

Nik: Yeah, I see it very differently.

I see it like when you say 20,
30,000 per second, it means for me it

becomes 2,000, 3,000 or 200, 300 per
second if we talk about really long

transaction happening in parallel
because they block vacuum, uh, work.

We cannot delete the tuples, and at
very high throughput with shorter

transactions will be-become the same
if you have much lower throughput,

Peter: Yeah, which is why
our recommendation is isolate

the DBOS system database

from the database we're using, we're
using long-running transactions.

These don't have to be in the same, in
the… These don't have to be in the

same Postgres database, and they probably
shouldn't be, especially for long-running

transactions that could block Autovacuum

Nik: Yeah, this is what my… Yeah,
th-this is exactly what I was asking.

It's not like like requirement, but
it definitely if you have long-running

transactions or any other explain
horizon blockers performance of

DB-DBOS will, will suffer eventually.

Peter: Yeah, but these don't, these
can be isolated from each other.

They don't have to run in the same
place, and they probably shouldn't run

in the same place, especially at scale

Nik: This is interesting.

We have a lot of common things.

I agree with putting everything
to Postgres open source.

I w- didn't mention that the thing
you're building is open source, which

is great, and like this is super cool.

And and like we don't need
extra things on critical path.

That's great.

Also I guess you have self-served
option as well, right?

If it is open source, you
can self-host yourself.

Like freedom, right?

It's great.

And you run also y-you
can run DBOS in, on RDS.

Absolute fr-freedom.

But when you say let's split it,
it's against the first principle.

Like first principle, let's
have it everything in one place.

Because if we have, this thing
and we have other tables, we can

organize code right inside database.

Sometimes it's a bad idea if
it's very long, if CPU-consuming.

But if it's something simple, I
want to update everything inside,

like using triggers, PostgreSQL
code functions, and so on.

I don't want to split.

But once I have database, say terabyte in
size, and I start some analytical queries

right there because I'm not I'm not grown
enough to have ClickHouse or something.

I have only Postgres, right?

Those long-running queries
will take time, right?

And in this case updates and data, MVCC
in Postgres actually is not a good thing

to deal with, as we all know, right?

This is I'm j- I'm asking because, like
you m- you know that I'm trying to revive

this PGQ from Skype, which is bloat-free.

No, almost no updates.

For event table, there
are no updates there.

But I don't see how it will… I
don't see clearly how it will be

used to implement workflows yet.

But for many tasks, it's great, and we
don't have problems with bloat there.

And I'm trying to understand
how our operate properly.

Like in, in case of updates, we
need to establish some rules.

Like for example, we should stop
allowing long-running transactions.

I'm I'm glad I participated in
building transaction timeout.

So we set transaction
timeout, say like 10 minutes.

That's it.

And I'm very curious, like maybe
there is a way to define some

boundaries and rules when it's time
to move away and to like to split to

different Postgres cluster, right?

To split these workloads.

M- And there are no good rules yet.

Th-This is a problem

Peter: One is that you don't have to split
multiple Postgres clusters or servers.

You can, if I remember how Autovacuum
works correctly, it is isolated across

multiple databases on the same cluster.

Nik: Logical databases.

Peter: Yes.

So like the xmin horizon
is set per database.

So so you don't have to split
your physical infrastructure.

You can share physical infrastructure

Nik: Yeah that's a good point.

I agree.

Still, I feel like it would be great
to have some benchmarks or something to

define when it's time, when it's time
to split, when it's fine, and which

transactions should be disallowed.

I think it-- you're very right.

If you have 30,000 workflows per
second it-- your requirements become

much stricter in terms of what means
long-running transaction, right?

How much xmin horizon is lagging.

Anyway, thank you for
diving into this topic.

I'm also trying to learn this myself

Qian: Yeah.

I think that's actually a good, really
good point because you are right that

if you put… Y- there are benefits
of co-locating the workflow tables

with your actual application data.

One thing we were talking about
in some of our blog posts is this

transactional outbox pattern.

If you say someone, like we updated
the customer data, now we want to

kickstart some of the workflows,
then by co-locating them in the same

database, you can do it in a trigger.

Say, whenever you update this customer
data, automatically start or enqueue

a DBOS workflow in the same database,
and that's really convenient and

that, that's a way that you can
guarantee whenever there's updates

in this table, you'll kickstart a
workflow that is durable, that will,

that DBOS will guarantee will finish.

And that's a pattern that is
really difficult to do with,

say, like external orchestrator
because there will always be a gap.

And by co-locating in the same database,
there's a way to close this gap.

But everything has a trade-off.

Like you said, if you put in the
same database, you have to be really

careful about what you, what queries
you put in that database, like how

to avoid head of line blocking, how
to avoid bloating in that database.

So I think that's a topic we're
continuously, benchmarking and

also by talking to users and see
what kind of workflows and what

kind of workloads they're running.

And we're, we are also trying to
categorize like a more detailed,

guidelines for people as well.

Nik: Yeah, interesting.

But any-anyway I'm glad that this, this
topic is becoming more and more popular

and other companies started to join
t- like M-Microsoft recently, right?

With pgDurable.

What do you think about that system?

Qian: Yeah.

So this is great.

I think pgDurable is mostly
for pure SQLs, right?

It's, it doesn't really work for
if you want to send a email or

interact with external systems.

So we'll definitely, take a look and
see if there are any ways to collaborate

or borrow ideas from pgDurable.

But I think in general it's
another validation of the Postgres

Nik: yeah.

yeah.

Everything more and more people
start looking in the same direction.

I think yeah, the topic is warming up.

I agree with you here.

Also, there was a system called
Absurd, which got my attention.

Single SQL file as well.

It… I think it was
released last November.

And yeah.

I noticed in pgdurable is that the
tendency to have a SQL PL, PostgreSQL

code quite ef-ephemeral way.

And this is w- like you're just
passing it as a parameter, like

to lump the functions, right?

And this concerns me because I want
my code still to be in Git, right?

To be versionable, fix,
reviewable, and so on, right?

So this leads me to question, what
are the best use cases, case studies

you ha- you had already, you where
you observed debor… I'm sorry,

like I pronounce it DBOS for myself.

I don't know why.

What are the best use case studies,
use cases you have, like where

it shines in terms of how like,
like it suits really well, right?

Sending emails or like processing orders
in e-commerce or something like this.

What are the best cases?

Peter: Yeah.

the thing about workflows is that
they're really powerful f- they're

really powerful for a lot of things.

So of course, the, the really
hype new use case we see is AI

agents and operations around them.

So people using workflows both to
make their actual agents durable.

This is particularly important for
long-running interactive agents that

might, talk to a person or do multiple
rounds of conversation with a person

or require approvals for a person
over the course of hours or days.

Similarly, data pipelines like document
processing pipelines for agents often

require durability, and, you know,
workflows can really help there.

So those are kind of
the, the new use cases.

And then there's this whole host of
traditional business workflows that

have always needed durability and
where Postgres backed durability

just makes it easier to provide it.

And that's things like logistics.

It's things like, like
a- like lab automation.

It's things like new user registration,
making sure new users are properly

updated in every single one of your
cor- of your corporation's databases.

So we see this really broad spectrum
of use cases for workflows because

really any important operation that
touches customer data, touches important

data, it probably should be a workflow
if you're really operating at scale

and you really care about reliability

Qian: I think in summary it's we
really view Postgres as the central

orchestrator to connect things together.

You can connect external systems
to your local database, or you

can do event processing to connect
multiple systems like your other

databases or your email server,
payment system, all sorts of things.

So we re- try to make sure your
Postgres is the glue for everything.

Yeah

Nik: And speaking of Postgres
maybe last question from me.

So now Postgres 19 beta 2
just released, Postgres 20

development has started already.

What what do you think should be changed
in Postgres or brought to Postgres to make

the product you are building even better?

Peter: I think we do have to go…
I would like to go back to notify.

Removing that global lock is not easy,
and I'm not sure if it's doable with the

way notify is currently implemented, but
that area of the code hasn't really been

worked on much since Postgres 9 and,
maybe there's something that could be done

there to remove the footguns from notify.

And even if you don't remove them,
I think notify's really complicated

performance characteristics should
probably be, should probably be actually

documented instead of having to figure
this out like through by… I mean,

we figured this out basically by,
by reading the Postgres source code.

That, that's one thing.

Qian: Yeah, I think another thing
is about, like, how to ho- better

s- even better support for larger
larger blobs or larger columns.

when you-- when people are using DBOS,
sometimes they want to checkpoint, say, a

PDF or something as part of the output.

And still I know Postgres has some good
support for larger blobs right now.

But I think most of the time, we still
recommend people to say "Upload your

large documents to S3, then store a
pointer in the database." But that still

leaves, a little bit gap in the durable
checkpointing, because there could be

the case where you upload something
but you don't checkpoint or vice versa.

It'll be great if we can really put
more, like, larger data into Postgres.

Michael: There is a nice change,
I think, coming in 19 around that

in that the, I think the default
TOAST compression is changing from

like a, an older PGLZ, like a
Postgres specific compression

algorithm to, I think it's LZ4.

but yeah only minor on, on
the grand scheme of things.

nice one

Nik: Yeah.

And speaking of notify, I think, yes,
global lock is still there, not removed.

I remember some work
last year in PostgreSQL

Peter: There was work in 19 on
improving the performance of

broadcast, of the notify broadcast
when not all listeners are listening

for all messages, which is great.

That's a real performance improvement.

The global lock is much harder to get rid
of, and I'm not sure if it's possible.

But that is…

Nik: work on removing that lock.

I remember that, that work.

I also remember I just founded my
email to document, as you said,

the limitations from July 10, 2025.

Nobody answered my email.

So there is a patch proposed.

Let's document it at least.

Yeah,

Peter: Yeah, the I, certainly 19 doesn't
include any work on the global lock.

Maybe there's some in
development for 20, I'm not

Nik: Yeah.

Yeah.

Yeah.

It's worth checking.

And that documentation should be changed.

I agree with you.

So maybe someone listening to us will
go and comment on that proposal, and

we will get it documented at least.

Great.

Thank you so much.

I enjoyed it a lot.

I wish you all the best
with this wonderful product.

I'm going to try it in a couple
of places, and I encourage

our listeners to try it out.

Especially it's open source, so
it's so easy to start, right?

Thank you

Michael: Nice one.

Thanks very much, both of you

Peter: Yeah, thanks a lot

Nik: Yeah, have a great week.

Qian: You

too.

Nik: Bye-bye