Michael: Hello and welcome to Postgres.FM,
a weekly show about

all things PostgreSQL.

I am Michael, founder of pgMustard
and I'm joined as usual

by Nikolay, founder of Postgres AI.

Hey Nik, how you doing?

Nikolay: Hi, Michael.

I'm doing great.

How are you?

Michael: Good.

Thank you.

And this week is launch week or
was maybe if people are listening

in the future.

Nikolay: Yeah, I can tell you,
I can assure you people are listening

to us in the future because we
are not live.

Michael: Yeah, of course.

So yes, so launch week for your
company, Postgres AI.

Nikolay: Yeah, for the first time.

I'm excited.

Yeah.

So we decided enough observing
how others do it.

First of all, Supabase, they just
had launch week before this.

I think number 15 already.

So many.

And since our team has grown and
we had stuff accumulated to

be released, so instead of releasing
it earlier, I said let's

just accumulate a little bit and
release in 1 week.

It's like sprint 5 days in a row.

And we actually, we are releasing
more, we, I hope we released

by the time people are listening,
I hope we released more than

5 things.

So we have also concept of extra
bonus releases.

And we started on Monday, we started
with DBLab version 4, which

actually took 2 years to develop.

Michael: Yeah, that's a long time.

Nikolay: Yeah, finally we polished
it and many users already

installed it.

I mean release candidates.

We needed to have 5 release candidates.

Michael: That's more
than Postgres.

Nikolay: Yeah, yeah.

Well, because a lot of new problems
appeared because we changed

a lot there.

Yeah, but it's another story.

Today we talk about our second
big release, which is called Postgres AI

Monitoring.

Michael: Yeah, so this is 1 you're
doing, so we're recording

on Monday.

This is 1 you're doing tomorrow.

So what is it?

What are you up to?

Nikolay: Yes, we are doing this tomorrow.

Everything is ready already.

So I'm pretty confident we are doing this.

Yeah.

And only a few people know that we are releasing this kind of

monitoring.

And it's called Postgres AI monitoring, but right now there is

a little AI there.

And this is intentional.

Let me explain this first, because of course we had a short chat

before recording and you noticed this discrepancy.

We use AI a lot.

Every day, for example, in my team, it's prohibited to do anything

without AI-first.

It's prohibited to write code.

It's already so.

Well, you can write code, but only if your AI tools failed.

And first people were resistant to this.

Some people said, like, what the heck, I don't want it.

It's bad, it's poor quality.

But in my opinion, the Anthropic release of Cloud 4 a couple

of months ago, it was a game changer.

Quality already very acceptable in many cases.

Not always, sometimes, especially if you are expert in some narrow

topic, you understand problems which were never discussed publicly.

In this case, of course, you will see AI will be wrong 100%.

We have these cases.

For example, a perfect example is our guests last week.

They experienced some problem which was not covered by public

discussions.

Obviously, in such cases, if you just apply AI in a straightforward

way, it will hallucinate some bullshit.

But if you apply AI to brainstorm some ideas and ground it, pre-ground,

post-ground with materials, reliable materials like source code,

documentation, information from reliable sources and experiments,

post-grounding, in this case it's different.

You can explore, behave, reproduce it, and this is the path we

chose.

Not just this problem, take AI and create, I don't know, like

change requests somewhere.

This won't work.

Ungrounded approach won't work.

So this is 1 point.

Another point is that 6, 7 years I keep talking, I avoid building

monitoring system.

For example, when I met with Lukas Fittl in San Francisco 1

day a few years ago, I told him this, like, you're building great

monitoring system, pganalyze.

We build some checkup because we need for our customers to find

problems and then to start exploring them, experimenting.

Our key strength of Postgres AI
team is experiments.

That's why we built DBLab to simplify
specific type of experiments

and benchmarks and so on.

And I mentioned checkups and somehow
pganalyze has checkup section

since then, but it's a different
story.

And I said I'm not going to build
monitoring system, but it was

inevitably like I was pushed somehow
to this point where it's

inevitable that we need to do it
because no other system, and

not only open source system, not
other system is providing abilities

we need.

We need several specific abilities
with our clients.

We always show it.

And some of them is for troubleshooting.

So like high, very high level analysis
of Postgres components.

And then you can understand where
you can dive deeper.

And actually, I remember great
talks from Percona, from Zaitsev,

and so on.

And I saw some bits of good stuff
here and there, like

Percona Monitoring and Management,
or how is it called?

Michael: PMM.

Nikolay: Yeah.

Yeah, yeah, yeah.

And things like use and 4 golden
signals approaches like methodologies.

Some great pieces there.

But then you see something is really
missing.

Like For example, proper query
analysis.

We talked about it so many times.

Comprehensive analysis using pg_stat_statements, pg_stat_kcache,

pg_wait_sampling, top-down, single
query with query texts with

proper characteristics.

And We didn't see any system like
this.

A few years ago, we started maintaining
so-called pgwatch2 Postgres AI

edition, which was a fork of pgwatch2
with our metrics and dashboards.

But then Pavlo Golub decided to
rewrite pgwatch.

So we started lagging in terms
of this vehicle we used as pgwatch

itself.

So time has come and we realized,
okay, and we experimented a

lot.

We actually started with InfluxDB.

It was insane.

We eliminated it.

We used TimescaleDB.

And then we had a big thinking
process.

Should we stay with PureSQL and
TimescaleDB, or we should use

PromQL?

We decided to use PromQL, but with
some nuances.

We still have Postgres inside.

So we have both Prometheus and
Postgres.

So the idea was we need to build
a new monitoring system, which

will be part of our process.

And in consulting, we had maybe 30 clients over the last 1 year

and a half.

We had a few dozens of clients, right?

And to all of them, we installed always our pgwatch2 Postgres AI

edition, and also used

postgres-checkup tool, old tool we

always used, to generate reports in static form, like markdown

or HTML PDF.

So this time we said, okay, we finally do monitoring, but the

main part of it will be open source.

So monitoring itself is open source, but it's a part of bigger,

bigger picture where we perform health checks and help clients

proactively using AI, of course.

Michael: So I'm going to ask your question always.

When you say open source, what do you mean?

Nikolay: License is Apache 2.0.

As I remember, pgwatch is a BSD or Postgres license?

1 of these 2, but our dashboards are Apache 2.0, so anyone can

use it.

Majority of that is...

So we couldn't use a so-called traditional Postgres exporter

for Prometheus because it's so much stuff and it's really slow

to put pieces there.

So basically we have our own exporter inside, I mean, metrics

and dashboards.

And it's Prometheus.

Why Prometheus?

Because we just see that in many cases, well, several reasons.

1 of the key reasons is that we had already 2, actually 3 companies

where we, in consulting mode, we contributed a lot to implementing

our approach to their own monitoring system.

So this is why we have confidence this monitoring system is great.

And usually, bigger companies, they have some stack.

Some people use Datadog, but you know, if you have 100 terabytes

of data, or say, 1000 clusters, like microservices, Datadog is

super expensive.

Some people still keep paying.

But it's not only expensive, it has blind spots in terms of Postgres

monitoring.

A lot of them, for example, pg_stat_statements, they have some

not all metrics.

And for essential metrics, they, for example, can give you per

call, but not per second, or per second, but not per call.

There is no completeness there.

If you check carefully, so they differentiate in 1 dimension,

but not another.

For another metric, it's vice versa.

So it lacks many things.

And also how dashboards are organized.

I don't understand why people put Tuple stats on the front page.

It's not the main metric to observe.

Not the main metric.

It's not the number 1 thing you want to see when you have an

incident and need to troubleshoot.

And people don't put queries per second because Postgres doesn't

have queries per second.

But we can extract it from previous assignments with some asterisk

that it's approximate because some things can be lost.

Sometimes we see queries per second are lower than transactions

per second, but it cannot be so.

Right?

Yeah.

In fact, because we don't see full picture, but in most cases

it's vice versa.

So we see vice versa.

So this monitoring is open source fully, and it can be used in

pieces if your organization already has Prometheus-based something.

But lacks a systematic approach for dashboards.

This is where we can, I mean, you can take this and improve,

take pieces, maybe contribute back?

Also good idea to contribute back always.

And I still think, although I personally invested like more than

5 years already to this.

First few years I wanted to say, no, no, I'm not going to do

it.

But then I was forced.

Well, well, life sometimes force you to do something right.

And now this is my like, I resurrected to this idea, like, okay,

I'm doing this.

And yeah, and not only I, we have really super experienced members

in our team, who also spent a lot of time thinking and contributing

to this.

It's not only my own effort.

So we have several very good team members who contributed a lot.

So we decided to use PromQL also because of this, because it's

more compatible with enterprise systems.

Michael: Yeah, you mentioned using Prometheus, but also Postgres.

Nikolay: But also Postgres, because in Prometheus there is a

problem with query texts.

In query analysis, you do want, like you have to just have statements,

But in Prometheus, it's not normal to get query and store it.

It's very inefficient.

So you need secondary storage, and we chose Postgres, obviously.

Basically we have 2 data stores.

1 is the main 1, Prometheus.

And the second 1 is...

Actually we are debating right now.

We probably will...

Prometheus, it will be Mimir.

We also thought about Victoria Metrics.

So it's still...

We will switch in the next few releases to something else.

And for query texts, it's Postgres, but it's a small 1.

Because you know, by default, pg_stat_statements for each node,

5000 statements.

And do we track all of them?

I need to check.

Maybe no.

So 1 of the things we noticed, we had cases with our previous

monitoring, that sometimes people have millions of objects in

the database.

And this is really requires optimization.

And we made this optimization for really large cases.

I remember our checkup actually also had cases with 100,000

indexes.

In case of monitoring pgwatch2, it was previous version, basically,

right?

Our Postgres AI edition, we had cases with more than 1 million

indexes and hundreds of thousands of tables.

Bloat analysis was not working.

But like, it's and this is interesting and a separate question,

how to perform index maintenance in such a situation, for example.

But

Michael: it's also a weakness of pg_stat_statements, right?

Because often, I don't know if this is the case always, but often

when you see that number of objects, it's because somebody's

doing per tenant, but by schema.

So a lot of the tables are the same dozen or several dozen tables

and indexes.

And then pg_stat_statements kind of falls flat on its face because

it sees objects as unique and it tracks them all individually.

So you can't look at.

It's flooded.

Yeah, and 5000 is not enough.

Nikolay: It's not aggregating properly.

Michael: Exactly, yeah.

Nikolay: Yeah, we discussed this recently also, there is a problem

with temporary tables, for example,

Michael: sometimes

Nikolay: it's like the same schema, same table, but somehow you

end up having different queries, 2 different queries and pages

are set in the same.

I would personally prefer some control how aggregation is done.

There is a compute_query_id mechanism.

So basically it would be great to define a rule, for example,

ignore namespace, ignore schema name, right?

Michael: Yeah, wow.

Yeah, interesting idea.

Nikolay: Yeah, maybe.

Michael: Because the object, the other table names will probably,

for most people, the table names
would stay the same, it's the

schema name that would change.

Nikolay: When analyzing logs, I
still prefer, I know since Postgres

16 we have query ID in the logs.

We can't have, we need to enable
it for auto-explain, but I still

often prefer Lukas Fittl's, already
mentioned him, his second

time, his great library libpgquery
and Qt fingerprints.

I know it works purely based on
syntax, no OIDs from pg_class

or something, so I know how it
will aggregate.

And it's mostly often enough.

Michael: Do you know if it can
handle that?

The different schemas, same object
name?

Nikolay: Well no, I think it distinguishes
them.

If query has fully qualified names
involving schema name, I think

it will produce different fingerprints.

But I guess it's easier to tweak
it than to try to change.

Michael: Maybe.

Nikolay: pg_stat_statements approach
and so on.

I don't know.

I just feel, yeah, it would be
great to have some more flexible,

some default path, but some exotic
paths to tune it, how we aggregate.

Because otherwise you end up with
5000 slots in pg_stat_statements

by default, And then maybe 5 different
queries, each 1000 times,

right?

But just with some differences.

You know, the in problem, I don't
know, was it fixed already?

I know, for example, the lipoge
query, regardless of the members

of in list, it will produce the
same fingerprint.

Michael: Nice.

Okay.

I don't know if the feature set
statements got fixed, but yeah,

you're right.

If there's like the difference
in having 1 in list versus 2 versus

3, they were all considered unique.

Nikolay: Some people implemented
some regexps to convert it before

calculating something.

Usually you want, if you have in-listed,
regardless of number

of members in the list, it should
be considered the same.

Semantically it's the same.

2 members, 3 members, it doesn't
matter.

But pg_stat_statements, that was until, like,
I thought I saw some discussion.

I like my memory tricks me sometimes.

Maybe it

Michael: was definitely been discussions.

I just don't know if there's any
commit.

I think gut feeling is it might
be in a very recent version,

but I'm not I'm not 100% sure without
checking the but the like,

for example, any if you if you
if you transform it into a query

with any, you're going to get the
same entry in pg_stat_statements

every time.

So why should it be different with
in lists?

Like it's, it doesn't make sense.

Yeah.

Although they could have different
plans, like at a certain point

it could flip the plan, but that's
true of any query.

That's true of any querying.

Nikolay: How work, how they work
with not in and nulls, right.

Remember this danger, right.

So yeah, let's discuss.

So yeah, open source, Apache 2.0,
open source components, PG

version 3 as like engine for this.

So we just,

Michael: pgwatch

Nikolay: pgwatch version
3, which was written by Pavlo

Golub, CYBERTEC, great work, a
lot of optimizations, a lot of

things like how we control, they
call it sinks, right?

Like how data streams are controlled
and so on.

And what else?

Very fresh Grafana, obviously,
right?

And there we have several interesting
tricks there.

So where does this dashboards shine?

If we talk about only dashboards,
as I said, it's a part of bigger

vision, but just talking about
dashboards and manual use of them.

First dashboard is the troubleshooting
dashboard.

You have just 1 minute and you
need to understand where to dig

further, right?

We use 4 golden signals there.

So we have throughput, latencies,
errors, what else?

Saturation, right?

Michael: I didn't see that 1.

Dashboard, okay, dashboards.

Nikolay: Dashboard number 1.

Michael: Active session history?

Nikolay: Yeah.

No, active session history, This
is where it starts.

But it's a single node performance
overview, number 1.

Active session history, in our
opinion, we're convinced after

many years observing how RDS does
it, others are also convinced.

Wait event analysis is the king
in monitoring.

If you want 1 single graph or how
to say chart, this is to understand

like quickly performance and what's
wrong.

This is what you need to look at.

That's why we put it on the very
first place on the very first

dashboard.

But it's only very high level,
like 30,000 feet view.

So only wait event types, not wait
events.

No query IDs, not particular wait
events, only wait event types.

So it's very, like, is it like
CPU or I or something?

And CPU, we mark with asterisk
because it's not only CPU we know

sometimes some parts of the code
still are not covered by by

wait events.

So technically, it can be CPU or
unknown wait event.

Right?

Yeah.

But and CPU is green is as everywhere
is in performance insights,

PASH Viewer, anywhere.

So then we have very high level
pieces to understand workload

and various Postgres components.

And we follow 4 golden signals
there.

So latency, traffic, situation,
traffic throughput, right?

Situation errors.

Latencies, we extract from pg_stat_statements.

I think it's a great idea actually
to have it in the

pg_stat_database.

Probably there are reasons why
it's still not there.

You can extract it if you have
PgBouncer, sometimes I just check

PgBouncer logs manually.

To double check that I understand
latency properly.

But from pg_stat_statements, good
enough.

Of course, again, if you have a
situation like we just discussed,

and you see only the tip of the
iceberg in those pg_stat_statements.max

entries of pg_stat_statements,
5000 by default, then it will

be a wrong value.

So I think, yeah, we, we, we, we
understand this, right?

So it's like best efforts.

It's kind

Michael: of an edge case as well,
right?

Yeah.

It's not many people that have
hundreds of thousands of objects.

Nikolay: Yeah.

Yeah.

And throughput, we like in various
ways, like calls per second,

transactions per second.

So these are key ones.

And we didn't put tuples per second,
as you can find in Datadog

and some other places.

Maybe we should, but I'm not convinced
this is super low level

throughput metric should be on
the very front.

Maybe actually, because I'm contradicting
with myself.

I'm always discussing logical replication.

I say on modern Intel and AMD processors,
single slot, Single

publication subscription, it can
saturate subscriber side single

threaded process at levels like
1000, 2000, 3000 tuples

per second, depending on various
factors.

So maybe tuples per second for
writes, maybe we should put it,

actually.

Good idea, maybe.

But this is very low level throughput.

So in database you can send throughput
in various levels.

Transactions, queries, well, tuples
next.

Is tuples next?

Or maybe rows?

Rows, tuples are different, right?

Well, buffers, yeah, yeah.

Actually, this is what we do have.

We have hits and reads and writes,
dirtied buffers.

All those buffers we put it right
on the front page, but only

high level, we don't distinguish
by query ID, this is the job

for different dashboard.

So you quickly can understand what's
happening here.

And of course, we translate it
to bytes, right?

So how many bytes per second of
buffer reads your buffer pool

is having right now?

And did we have spikes, right?

So yeah.

And actually, yeah, and errors.

Yeah.

So if we are errors, we will see
it in transaction analysis,

number of rollbacks.

We currently don't work with logs,
but it's in the roadmap, of

course.

So we extract errors from logs,
aggregate, and so on.

I still have hope that a tiny but
very useful extension called

logerrors.

1 day we'll make it into engine
and we will have some pg_stat_errors

or something.

It would be great to have some
error code and how many times

it occurred, right?

Basic accumulative metric and we
could start observing them in

normal way, exporting and then
alerting.

Speaking of alerts, it's also in
roadmap, definitely.

We have very careful approach because
talking to different customers,

we see very different expectations,
very like opposite sometimes.

So yeah, I will discuss it.

Yeah.

In the past.

Michael: Well, I think it's interesting.

I think alerting is fascinating
and like balancing risks of false

positives versus risks of false
negatives.

Like I think it's a really hard
problem.

Nikolay: It's a hard problem.

Michael: And configurationally,
People always just end up making

everything configurable and it's...

And picking defaults is hard or
like...

Anyway, it's not envious of people
that make monitoring tools.

They quite like making

Nikolay: it simple.

Yeah, I think we will avoid a single
default set, set of defaults,

and we will just give some, basically,
a questionnaire.

What matters for you more?

More like this or that?

And based on that we adjust the
setting right away.

Michael: Like when people first
install it, there's like the

old school wizards that you go
through and answer a bunch

Nikolay: of questions

Michael: and then...

Nikolay: Old school Typeform, you
know?

I'm joking, I'm joking.

Typeform is already old school.

Because I see dental clinics use
Typeform already.

It's not cool already, it's old,
ancient.

I like actually, I love Typeform.

I love the fact that you can use
only keyboard, no mouse involved,

navigate all the questions, answer
them.

So anyway, yeah, alerts is a separate
topic, it's in the roadmap,

not right now.

We also paid very careful attention
to metrics we display.

We had many debates about this,
really.

For example, we just talked about,
for example, buffer reads

per second.

We translate them to bytes, so
megabytes, mebibytes, gibibytes.

We use lowercase I in the middle
to emphasize it's gibibytes,

mebibytes.

So if we display it on X's Y, is
it megabytes or mebibytes

per second?

And I don't see consistency in
various monitoring systems.

So I managed to convince we should
display maybe bytes per second.

And to convince, I just said, okay,
actually, you know what?

We have all these metrics, it's
differentiated by time.

So if you display a raw metric,
it will be constantly growing.

It's a cumulative metric, it's
just a number which always grows

until you reset it.

So since you differentiate it by
time, it becomes already spiky

or some kind of normal behavior.

So it should be maybe by second.

But also, I said, notice we develop
additional differentiating

on the right side, almost always
you can find a chart which is

displaying the same metric per
call, average per call.

Michael: So flipping instead of
per second, per call, per query.

Nikolay: Exactly.

So if it's, for example, execution
time per call, it's average

execution part of the latency.

Michael: That 1 you can't do.

Nikolay: Total exact time per second
you can do.

This is the key metric, this is
the central metric in our approach.

It's measured in seconds per second.

We discussed this, yes.

I'm glad we're repeating this topic.

This is key metric.

Michael: Yeah, me too.

I've clearly forgotten it.

Nikolay: Seconds per second is
a key metric.

My favorite metric.

It shows you how many seconds database
spends to execute your

queries per each second.

Michael: And then do you have to
divide it by cause to like understand

you to like say shut up?

No, no, no,

Nikolay: no, no, no, no, no.

Wait, wait, wait, wait.

It's 2 different things.

We have some metric, for example,
number of buffers, which is

also always growing.

Across all query ideas, here it's
top level.

We don't, we just summarize all
of them, all 5000, right?

Or we have total exact time, right?

These are 2 raw metrics.

They are always growing, growing,
growing.

You can let Grafana, basically
PromQL using irate, You can let

it differentiate by time.

We don't use mean exact time because
we can have it from different

place.

Or you can divide it by calls.

If you divide by calls, you have
basically the same value as

the mean exact time, but not Postgres
calculated it, but PromQL

calculated it.

But if you divide it by time, it's
differentiating by time.

You will see how it changes over
time.

Every second we spend, for example,
10 seconds, give us at least

10 cores for this workload.

It's very roughly, it doesn't fare
because this time can include

waiting on lock acquisition, or
I.O.

It's not true CPU, user or system
time.

But roughly, it gives you a good
understanding, what kind of,

like the scale of your workload.

If it's 10, don't run it on 2 core
machines.

Michael: Yeah.

Well, more to the point, is it
growing over time or are we reaching

some, you know, when are the spikes,
that kind of thing.

Yeah.

Interesting.

Nikolay: And if you divide it by
cost, you have average.

So that's why we say metric slash
S per second.

And if it's seconds per second,
we transparently say S slash

S.

Seconds per second.

Why not?

Michael: Well, I just think it
hurts my head even thinking about

it.

I think there's all sometimes a
case in products for being accurate

and sometimes the case of being
simple or like easy to understand

at first glance, and it depends
who you're building for, right?

Like accuracy is extremely helpful,
especially for deep experts.

They want to know precisely what
happened all the time, but anything

that could be distracting for folks
who don't necessarily know

what they're doing.

Nikolay: That's a great point.

Michael: Yeah.

So, or it was the pre you've answered
pretty much all the questions

I had already, except for who is
your, I know yourselves are

your ideal user, but like, who
is your ideal user for this

Nikolay: product?

99% ourselves.

We do it for our own work, for
consulting, but we see it beneficial

to publish it as open source, so
first of all, our clients could

verify what we are collecting.

Even Datadog publishes exporters,
so it's fair.

But for us, these dashboards, it's
not final product.

By the time we publish this release,
it's already obvious.

We already included this to workflow,
where meta collection is

step number 1 of big step number
1.

It's just first half of the step
number 1.

Dashboards is step number 1.

And then API is exporting dashboards
in form of static tables.

Actually we didn't get there, but
our dashboard with query analysis

includes not only graph view charts,
but also table view like

our postgres-checkup has had.

And table view is super beneficial
because you can compare for

each query ID, you can see many
different metrics in 1 row.

You can understand much better
looking at very compact form of

representation.

It's for experts, it's not for
non-experts.

Regular Backend engineer needs
some time to understand how to

work with this.

It's possible.

We did it, but it really needs
some time.

But since I said this is big step
number 1, these dashboards

and charts and tables and everything.

Second step is analysis by experts,
And experts can be humans

or LLM.

For efficient analysis, you need
to transform raw observations,

we call it, we transform observations
to conclusions.

You have some transformations where
it's quite like we did with

buffers.

Okay, 1000 buffers.

Even for experts, it takes some
time to understand is it big

or not.

But then we multiply it by 8 KB,
and we have 8000 kB, roughly

8 MB.

Not exactly, but roughly.

We say, oh, okay, this I can feel already.

I know how long does it take to get it from disk, how much of

memory it takes.

I already feel it.

My point, my hypothesis in what we are doing is that both experts

and LLMs work better with this transformed raw data.

If you present it better, This is what we do on second stage.

Second stage, we've transformed observations very raw to conclusions,

but we don't add thinking there.

Maybe a little bit, you know, a little bit.

Not debatable type of thinking, quite like straightforward logic,

you know.

Like We can calculate percentage, for example.

This query ID in buffer hits takes like 25%.

It's very noticeable.

1 fourth of whole hits traffic.

1 fourth of the whole hits traffic.

Michael: So, yeah, like, it sounds like the kind of arithmetic

that's difficult to do on the fly accurately when you're in the

middle of an outage or something.

Nikolay: It includes everything, like, arithmetic, like formulas.

It also can include some representation, transformations, like

put these numbers closer to each other.

Just by putting this, you highlight that they have some correlation

or something, you know?

Yeah.

Also, take specific charts.

This is already like methodology we develop over many years,

how we approach analysis of various problems.

But still, we don't put conclusions there because we just observe.

We just slightly transform what we see to better form for consumption

of bioexperts still and LLMs.

And final thing is recommendations and understanding problems.

Not only recommendations, first understanding problems.

So We see problems and we already start thinking.

This is already sometimes if you take 2 DBAs, they have 2 different

opinions.

What they see.

Here is already a tricky part.

But our idea is, okay, we have a brainstorm phase, a collection

of hypotheses.

And then, since we build very great lab environments, we have

thin clones, we have thick clones, we can have different tests,

and We can verify ideas on thin or thick clones.

Or maybe in synthetic, sometimes it's just before you're sending

bug report or raise some discussion on hackers, you need to have

minimal viable representation of your problem in synthetic form,

not to share personal data, right?

From production.

It should be minimal.

It's also a special type of art
in database operations to extract

from production some minimal stuff
and inject it.

And here is where LLM can be very
helpful, to iterate much faster.

Michael: I find this difficult
for blog posts as well, like coming

up with an example that you can
share.

Anyway, it's an art and very impressive
and takes way longer

than it looks.

Well, at least it does for me.

Nikolay: It should take less.

And I hope AI...

I don't hope AI.

AI is already improving this, but
I think we are on the edge

of massive improvements in this
area.

Michael: I really hope so.

I've tried it for a few, I've tried
using even Claude 4 with

limited success.

Nikolay: Yeah, I know it.

Yeah.

I know, I know some problems like
you just give up.

We spent whole last year experimenting
so much.

We still have it.

We still have credits from Google
Cloud, Gemini.

Gemini excels in some parts of
work, but not in all.

And of course, Claude is not also
not the final answer to everything.

But I see many, like, There is
a substantial improvement with

Claude 4.

It gives me a much more level of
excitement of how we can move

forward.

This is our whole system.

Right now, almost everywhere is
human, because we don't trust.

It will take a few years to start
trusting more.

Maybe LLMs should be...

Well, they don't stop.

They are improving.

Right?

And knowledge base is growing also.

Our knowledge base is growing,
which we use when we work with

LLMs to improve quality of answers.

This is how I see it as a component
in the beginning.

It can be used directly if you're
an expert, or you can spend

time understanding some documents
like to dive deeper to details,

and then not becoming an expert,
but already to start understanding

how things work.

And then we transform it to form
which is easier to move faster

basically, not to jump between
various pieces we already.

For specific analysis we compile
multiple reports to small report

which is for this specific analysis.

For example our checkup has a report
called Memory Analysis.

Memory Analysis, if you want to
tune memory, not only you need

to tune shared_buffers, maintenance
work_mem, work_mem, max_connections,

but also check what's happening
with your number of autovacuum

workers.

And also remember that in Postgres
17, before Postgres 17, even

if you raised maintenance work mem
beyond 1 gigabyte and put it

to like 8 gigabytes, thinking,
okay, I'm going to create or recreate

indexes only in 1 session.

But autovacuum workers, you have
them, you also raise number

of autovacuum max workers, you
raise it to 10, for example.

And you think, okay, I know that,
well, people don't know mostly,

but I also didn't know until recently.

But before 17, If autovacuum work mem
is minus 1 It means that

inheritance from maintenance work mem
but maintenance work mem

if you rest beyond 1 gigabyte It's
still autovacuum cannot use

more than 1 gigabyte.

But this was changed in Postgres
17, it can use more than 1

gigabyte right now.

So if you tuned it before and you
then upgrade, you can run out

of memory.

Suddenly, if you perform the vacuum
tuning in more, like we always

say, raise the number of workers,
give more memory to workers,

to backends which build indexes.

Now you upgrade to Postgres 17,
mathematics changes.

So we have a report which collects
all the pieces and involves

the version and knowledge about
these changes.

It requires some expert work.

And we should not live in a world
where this work is done manually.

So it should be done in an automated
fashion, collecting pieces,

analyzing things, not to miss some
important parts of the knowledge.

So LLMs there, definitely.

Michael: Well not definitely.

You can get versioned.

I'm intrigued as to how much you'd
end up using LLMs on this

journey, because things like the
version information, you can,

like, you can get directly from
Postgres, right?

You can run a query and find out
what version it is.

You don't have to work that out
or it's not something where like

a language model is going to be
beneficial.

And there's some, like, I don't
do monitoring tools, but doing

performance advice, it's like something
learn to like, try to

get good at.

And I thought we would end up wanting
to write bespoke advice

in lots of different cases, but
it's amazing how you can word

things so that they cover different
cases or conditionally show

different pieces of the advice in
different scenarios and piece

together the full response without risk of
hallucinations.

Nikolay: I agree with you.

That's why on the first 2 steps
we don't almost don't involve

LLM because we don't want them
to, to, to like, III cannot let

LLM to summarize the numbers.

I better use a regular calculator, right?

Michael: Yes.

There were some things previous generations of computing were

fantastic at.

Nikolay: That's why jumping from observations to conclusions,

we basically don't involve LLM.

We involve our methodologies we developed, how we analyze things,

and we just transform.

We don't need LLM there.

But when we have problems, First of all, we need to understand

which problems look most critical.

This, I believe, can be done at least preliminarily.

LLM can highlight what's the most critical.

Then most important, start hypothesizing about trying to understand

root cause, trying to prove it, build experiment to prove it,

and then how to solve it, build experiment how to solve it, prove

that it's like drive this very hard work of root cause analysis

and finding mitigation.

This can be done by LLMs because it can search similar results

on our knowledge base and internet, find ideas, and using thin

and thick clones and frameworks we build, it can start experimenting

to check and to prove, oh, this is what's happening here.

Of course, the problem we discussed last week with metronome

guys, It will take time for us to reach a point where our system

will perform full-fledged root cause analysis and experimentation

part.

But I would say it's like 1 or 2 years only.

We already had a lot of pieces built on this road.

Michael: IM.

Well, I look forward to discussing it again then.

Yeah.

Yeah.

Nice.

Nice 1, Nikolai.

Looking forward to it.

Good luck with the rest of your launch week as well.

Nikolay: Thank you.

Thank you.

Michael: In the past.

Nikolay: In the past.

Back to the future.

Or how to say.

Good.

Thank you so much.

Michael: Nice 1, take care.