Michael: Hello, and welcome to Postgres FM,
a weekly show about all things Postgres.

I am Michael founder of PG mustard.

This is Nikolay founder of Postgres AI.

Hey Nikolay What are we talking about today?

Nikolay: Hello, hello.

Let's talk about query optimization.

I think this is maybe the most interesting topic in in
the area of Pogs in general, but I found not everyone

is interested in it, but let's talk about it anyway.

Michael: Yeah, it's also, I guess, a, topic quite close to both of our hearts.

We've spent many years looking at this.

So hopefully we have some interesting things to add.

Nikolay: but let, let's make some boundaries.

Let's distinguish analysis of workload as a whole
and try attempts to find the, like the worst.

Best candidates for optimization versus single query optimization.

Let's talk about this, the, the second topic subtopic,

Michael: I've heard you differentiate between macro
performance analysis and micro performance analysis in the past.

So macro being system level and with, I guess, we're not
talking about that today and we're gonna look more at micro

once you've, worked out, there is a problematic query.

You know which one it is, how do you go from that to.

What can I do about it?

Nikolay: Right.

How, how to understand Is it good or bad in terms
of execution and how to read the query plan?

The comment, explain, which is the main tool here, right?

Let's talk about this.

Michael: Is there any other, is there anything else
before we dive into explain, are there any other

parts of it that we, we might need to cover as well?

Nikolay: well?

the, the things that explain doesn't cover, for example it, it won't show it.

Won't it won't tell you for example, CPU utilization, user CPU system CPU.

It won't tell you physical this Coyo.

how many operations at dis cloud, because PSUs doesn't see
them directly because Pogo works only with file system cash.

So I, your operations reason, his PSUs tell
tells you they are not necessarily from disk.

They are from page cash.

So this can get additional, not use and explain it would be good to.

These things inside, explain somehow like pist K cash extends pist statements.

So it'll be good to have something that would extend, explain,
but I don't, I'm not aware of such thing to exist and Yeah.

Michael: Nor me.

I'm not aware either, but I think we get clues about them in explain
don't we, we see some timing that can't be explained other otherwise

Nikolay: your timing human or,

Michael: Sorry.

No, I, I mean, let's say you mentioned CPU performance.

If we have an operation, that's not doing
much IO but it is taking a long, long time.

That's a clue that there might be something else going on.

Nikolay: right.

Right, right.

So in, in general, if we run even one query.

Theoretically, it might, it might be sense to use
things like P and C flame graph for one queer execution.

And it would augment the information that you
can extract from explain, analyze buffers.

But let's just discuss the basics.

Maybe if were to.

Michael: Sounds good and something so possibly even
explain versus explain analyze is a good place to start.

By explain we get the, the query plan that
often normally returns really quickly.

Roughly in the, the planning time of the query and then explain, analyze we
get the actual, well, it, it runs the query and returns performance data.

So we, we can see how much time was spent.

And if we ask for buffers, how much IO was done and all sorts of
other things as well, but it allows us to compare things like.

How much.

So the explain might tell us how many rows were being
expected to be returned at each stage and explain, analyze.

We can get the actual number of rows returned at each
stage and comparing the two can be really useful.

Nikolay: right, right.

And Yeah.

absolutely.

And also.

we discussed it some time ago that explained shows only one plan.

Sometimes you want to see multiple plans,
like second best candidate and so on.

Otherwise, like you should do some tricks to try to
guess what plan ahead on, on the plate when choosing.

Michael: That's a really good point.

And maybe even a good place to start in terms of using explain.

So the first thing you probably notice when you're looking
at explain for the first time is a lot of cost numbers.

These are an, an arbitrary unit that give you an idea of how expensive.

So it's a kind of an estimate of well, as they, as the cost numbers go up,
Postgres thinks it'll take longer to execute, but they're not in, they're

not an estimate of milliseconds or they're not in any real unit, but.

You can, you can then use a couple of different like, is it enable sex?

There's like some, there's some parameters you can use to affect those costs.

So you could maybe try and get the second
best plan by making the current plan.

Very expensive.

So if your query is currently doing the sequential
scan and you want to see if it could use an index and

it's just choosing not to, you can disable sex scans.

Well, it doesn't actually disable sex scans.

It just makes them incredibly expensive and you can get exactly.

So you might still see that in the

Nikolay: It's very, this trick is very helpful.

When two plans are very close to each other in terms of cost overall cost.

And if you disable six scan and see, or example index
scan and see different plan and see the cost is very

close, it gives you idea that we are on the edge, right.

Or.

We either crossed it recently because data
is changing or we are about to cross it.

And, and it's quite kind of dangerous.

But first of all, I would like to mention that
the plans, the plan a plan is a three right.

Cycles are not possible, which is important
because it could be possible actually.

But no, , I mean,

Michael: I think in the simplest cases yes, but with
CTE's you get some, like, you can get some strange things

can refer to the same CT more than once, for example.

But yeah, in the simplest cases,

Nikolay: but when it's.

already executed, that's still like a tree right.

Oh, interesting.

By the way.

Yes, Anyway, it's it is like very rough it's, it's a three
and it's when it it's printed, it's a three, but right.

Loops are possible inside, of course.

Right.

And important thing to understand that
regular metrics are as cost rose timing.

They are shown for one iteration inside the loop, but buffers,
they it's a sum of everything and it's, it can be confusing.

Sometimes right.

Michael: Yeah, well, let's go back to the tree.

I think that's really important.

And a few, a few things that aren't obvious when you're first
looking at them is that logically it's happening almost backwards.

So the first node that you see on the tree is the last one to be executed.

Whereas,

Nikolay: It grows from leaves to root

Michael: Exactly.

So kind of outside in like, kind of right to left a little bit.

And there's also some really important statistics, especially when you
use explain, analyze some really important statistics right at the bottom.

So some summary metrics like execution, time planning, time

Nikolay: Oh, they're printed separately from these three.

Right?

Right.

Michael: and like trigger time just in time compilation, each of these things.

Dominant sometimes they can be like where all of the time's going.

And if you have like a really long tree the general
recommendation is start kind of right to left.

But I also say check that bottom section because it
could be, you might not have to look through the entire

tree if you find out that your query's spending 90% a

Nikolay: And interesting additional point here is
that planning time can be sometimes very, very big.

I, I had cases when inspection of of the path of merge
joint led to huge scan and during the planning time, And.

Disabling merge scan helped, but it was not so obvious
at all in the beginning because if you don't notice

that planning time is seconds so insane, suddenly huge.

It's a surprise to you for you.

So checking, planning time also useful.

Michael: Yeah, same for time spent in triggers and time spent
just in time compilation as well for an some analytical queries.

you might consider it a relatively simple
query or quite, it should be quite fast.

But if the costs are overestimated a lot, sometimes the just
in time, compilation kicks in, spends several seconds thinking

it's saving your time, but then that's overall, you know, the,
the overall queries only a few milliseconds that's suboptimal.

Nikolay: Right.

And also since for August 13 planning time also like it's, if you
include buffers option, it'll show you buffers used for planning as well.

Right?

Michael: Yes.

And actually one other thing on planning time before we move on
from that, is that auto_explain doesn't include planning time.

So there you can't spot planning time issues in order
to explain other than I think we discussed this once.

Yeah.

You could, if you, I think we discussed it years ago or
ages ago, because it was a, it is probably the only use

case for logging the query time plus the order to explain.

Time, and then you could diff the two and it's probably planning time.

That's the difference.

But yeah, it's, it's a limitation, not as you say, not
super common that planning time's the dominant issue,

but when it is, it can be 90 plus percent easily.

Nikolay: Right.

It's it's gonna be unexpected.

This is the danger of it, right?

Michael: Yeah.

So right to.

In kind of inside out start at the bottom check.

The main statistics, you mentioned briefly
that some of the statistics are per, per loop.

So loops are quite, I think they're quite a confusing
topic when you're first getting used to it and.

Uh, Especially if there's, you know, 10,000 loops, you could easily miss that.

One of the statistics, it looks quite small, but
once you times it, by 10,000, it can be really big.

So things, examples are of course the costs and the
timing, but also things like rose removed by filter.

Sometimes people look out for those numbers.

If it, if it says one that's a per loop average and
actually 10,000 of those is SU suddenly not insignificant.

Nikolay: Right.

And those averages can be rough.

That's also like there's a, there is a
mistake that can be present, present there.

Michael: Well, especially around zero and one,
like, because some of the numbers are intes.

Yeah.

Ex so anybody that's wondering if.

Exactly.

It's rounded to the nearest integer.

And if it's less than nor 0.5 and gets rounded to zero, it doesn't
necessarily mean that there are zero which can be problematic.

Nikolay: Also, you know what, we discussed things that
probably there are many talks and many articles that are useful.

I think this podcast is not going to replace them.

We are trying to highlight problems which can be tricky for in the beginning.

Right.

And one of the things um, I think uh, We should mention tools as well.

Right?

So first of all, explain decom is the oldest one and
still very, very popular, maybe still the most popular

then explain decom, which is P P V two greatly improved.

Very good.

And of course, P master, which is commercial, which
you, you, you develop right worth checking all of them.

They are good.

Pro and cost for all of them.

But what I think is important to understand is some meta.

When you talk about query analysis of single query
analysis, we should say, okay, this is our plan.

better if it Was with execution and the ex like the
best, if it's execution collected with buffers and we

will discuss by the way, overhead a little bit later.

Right.

But when you ask someone to help with optimization, of
course, the first question will be showing the query itself.

Sometimes show me two plans.

Also, if, if there Was some change and we want
to understand why this change influenza the plan.

So we have two plans, but we need to have
a query like must have, we should have it.

But additionally, I think very important to get Pogs uh, settings.

for like enable sex can or random patch, cost patch, cost, all costs
work_mem as well, even though work ma'am is not inside planner.

Positive settings are grouped by in groups.

If you can select star from pg_settings and see group
names, and there is a whole group named planner.

Planner rule settings and work ma is not there,
but work ma influences planner decisions.

If you change work, ma'am plan can be different.

Right.

So right now my rule is like, let's take planner, settings plus work.

Ma'am maybe something else.

I, I like I'm interested to see if something else should be there as well.

So when we ask someone to help, we need to present a plan or to query.

Linear settings.

And I also believe that schema is important to present.

Like what kind of schema we had table indexes and probably statistics as well.

This is like whole picture to analyze what we had.

Imagine if the, all these tools collected these things
automatically, for example, how great it would be to.

Jump into like, I want to help someone
with optimization and I see whole picture.

I see query plan settings, the schema, not whole
schema, only part of it, which is involved in like,

which is with query deals with, and also statistics.

Maybe statistics is kind of tricky, but
This is what, basis for planner decisions.

This is what defines what plan will.

Oh, of course.

POG this version as well.

Because the different version of the plan can be different,
different nodes in the plan can, can be present depending on version.

So what do you think about this?

Like big whole picture.

I understand that none of tools collect this information
and none of, none of tools require users to present this

information, but it would be great to, to, to store it in history.

For example,

Michael: Yeah, there are some really interesting
tools that do some, some of that, but not all of it.

So there's a, a tool.

Started as a, my SQL tool, but they've, they've added
Postgres support called ever SQL that that asks for

things like the query and the schema and things like that.

And then does some static analysis, which is super interesting.

There's tools like PG analyze It's a monitoring tool and it's
starting to do some well, it's been for, for at least a couple

of years now doing more ad hoc performance, like query analysis,
fire, explain visualizations and has access to a lot of those.

A lot of that information already by the nature of being a monitoring tool.

But I think there's also this natural trade off between this macro analysis.

This, the, all of the information you can gather and the overhead of doing
so versus the amount of information you're willing to gather, to do a

like once, you know, a certain query is a problem you're willing to pay
a higher overhead because you only, you only need to gather that once.

Whereas if you want to do this all the time for every
query, I think there's a a slightly higher overhead.

So I think there's, there's some

tension.

There's

Nikolay: could do some hashing, for example, to track that like
statistics didn't change and Response experimenters didn't change.

We could just check it too automatically with hash.

Michael: Yeah, I think there's some super cool things here, but there's
also, there's a, there's a couple of different um, environments, right?

So in production, like the one, one key thing
is that let's say statistics is a great example.

Production's not the same as staging.

Like we could, we can make all of the data the same.

We can,

Nikolay: It depends.

Michael: sorry.

Yeah.

Production might not be the same as staging and
therefore like a statistics problem may not show up.

And if you are, let's say you're doing some development work.

It's, it's really tricky to, to reproduce all of those things.

So, and, and I think I'd also push back that some of those,
some of the most obvious things that can be a problem, it.

Most like, maybe even schema doesn't matter.

maybe even query doesn't matter if, if you see that somebody's doing a
sequential scan of 10 million MOS maybe probably it's parallel and they've

returned, they're doing a filter and it returns just one of those rows.

Without the query, we can tell that an

Nikolay: to suggest index right.

Michael: Exactly.

So.

There's a bunch of cases where you can give a, you can give some
pretty sensible advice without any of that extra information, but

definitely as it gets more complex, I think more of those things can
be more useful, but EV even in the index case, you know, you still

need, I think you still need context from the customer in terms of
trade offs to, you know, if they've, if this is a super high right

table, you might be less inclined to, to add an index than if it's not.

Or if the customer has certain requirements.

There's always gonna be a, there's always an, it depends.

Right?

There's always, I think whenever you're giving this kind of
advice, you have to be careful that for different customers,

different things would be sensible, I guess, except for, except for

Nikolay: Well, I understand that what I just described a
collection of all these pieces require requires a lot of efforts.

So that's why it should be automated.

But imagine if everything was collected automatically inside some tool.

we used an organization and stored historically, you understand?

Okay, we optimize this query and we try to
optimize this query and we know the whole context.

When we ask for help, we have all pieces and if
expert comes to help us, all pieces are present.

That's it?

It would be much easier to help.

Right.

Michael: Yeah.

And I think there are some, some interesting projects in this area.

I think you have you come across the one by Perona,
they they're doing a kind of a replacement to

Nikolay: well, it's about micro macroanalysis as well.

P monitor, right?

PTA monitor

Michael: Yes.

But I think they

do things like plan.

Nikolay: start statements or no,

Michael: Yes, but with additions, like I think
they let you track query plans per query.

So like, I think you could, for example, see if a plan has changed.

So it's that, that kind of thing.

With relatively low overhead, I think you
start to get a bit more of that information.

So when an expert comes along, hopefully this
is something already installed and already

Nikolay: This is old big discussion.

There is a current ongoing discussion in PJI hackers about
Brisco hackers, mailing list someone froms proposed adding

plan ID to PSA statements, triggering discussion one more time.

And this is, this would be great.

Of course.

We know that each query registered and P
statements can, might, might have multiple plan.

Depending on parameters used.

So when you optimize a query, very important thing.

I, I missed it.

My, in my list parameters you used, right?

Because different parameters may trigger the
planner to choose different plan.

Right?

So, so it's very, very important.

When optimize the query, we cannot say we optimize the query.

We, we, we must say.

We optimize a query for some parameters, and we need
to think about variations that we should expect on on

production and check them too, not just a single case, right?

So this, and this is tricky by the way.

Michael: Yeah.

So if anybody's wondering, this is like the simplest example of this
is let's say you have a column where 99% of the data is a single value.

And then one, the, the other 1% is millions of like unique values.

If you search for one of the unique values.

You might get an index scan.

If you search for the value that 99% of the
table is, then you should get a sequential scan.

That would be the, the optimal plan.

So that's the

Nikolay: Right.

This is classic example and even enable set, enable six can to.

Might not help to avoid six can in some cases.

And I also, I, I had a couple of times in my optimization activities.

I had the case when somebody provided my me a query without parameters
and I've checked the table, I saw, okay, what's the worst case.

And I started to optimize for the worst case and made bad decisions
because this worst case was, was never used in production.

so it's very, very interesting topic.

I definitely want to find some approach when we don't know which parameters.

We have, but we guess somehow, for example, if some
additional tool would analyzing statistics, this tool

would say, oh, take this set of parameter, this and this.

Like, this is most typical case.

This is like some kind of worst case and try to optimize for them.

This would be.

Michael: So, so yes, agree.

And I think there are some, I think, for example, auto explain.

Very old tool, but one thing it does really well is it
spits out the exact query that caused that slow plan.

And that's one way of getting at least the
extreme versions of the parameters that

Nikolay: Or just slow log.

If you have log duration statement above 500, 100 milliseconds which is
good, or at least a second or two, which is not so good, but also , fine.

Uh, You have examples of parameters which
trigger slow execution, but you don't see.

good parameter sets, which are not registered in this slow log.

When I say slow.

log I mean a part of single PSUs log, because PSUs has just one log.

It's a different, different discussion maybe.

And if log duration statement enabled to see the, the examples
with duration, but ought to explain it's even better with the plan.

Michael: Yeah.

Is this a good time to talk about overhead?

Nikolay: Yeah.

Let's talk about overhead.

So when you run, explain, analyze versus you run
query without any observability tooling, which

explain, explain analyze is, is observability tooling.

It, it adds a lot of details about query execution and planner decision.

Right.

But you can just run the query and see some timing, but
then run, explain, and license and see different timing.

Right?

You had a block post on this topic, but about auto
explain auto explain is also like a related question here.

Michael: Yeah, Andres had a really good blood post on the observer effect.

So I think, I think there's two, there's two cases that , not only
can explain, analyze, tell you that it, that it can be very accurate.

So it can be that it's roughly the same amount of
time as the, as running the query through your client.

It can be too high where it's adding overhead and it can be too low.

For the case where lots of data's being transmitted.

It doesn't transmit that data.

So it can even be faster than a query that would return the data.

So there's kind of three, three cases, two cases that are bad.

They're quite rare in my experience.

And they.

especially on modern hardware.

They don't show up that often.

And also they're not that problematic when
you're actually looking for the problem.

If, there's a relatively universal overhead added and you're
still looking for what's the slowest part, it's still probably

the same place, but yeah, let's explain why it happens.

uh, , it's doing, in order to measure timing.

There is some overhead

Nikolay: I would split it to three parts.

So sorry for interrupting.

I would split it to three parts first, when we
say explain, we just see the plan of decision.

We don't is good cure, query, nothing to discuss in terms of overhead here.

Right?

Well, there's, uh, The cost of planning, planning work,
but it's not overhead it's it's anyway, we need it.

But when we add analyze, There is overhead like we really execute
a query, but we need to measure things and see how manys in each

node were collected, everything like that and timing as well.

But they also can say buffers.

This, this is additional overhead.

And we also can say Tracko timing, which is a posts setting.

You can set dynamically I.

guess.

Right.

And you can see IO timing additionally printed
in by explain, by explaining lies here.

Right.

And this like three pieces of overhead.

Right?

What do you think about each of them?

Michael: Yes.

Well, as you mentioned, I did do a blog post on this
because I, I saw quite a few places where people would

really warn against or to explain with timing on.

There was really, there's a really strong warning
against it in the documentary, in the Postgres docs.

There's multiple monitoring tools that tell you if you
have, or to explain on, make sure you have timing off.

Nikolay: At the same time, I, I observe very heavily loaded
systems serving a lot of thousand, like hundred, hundred

thousand transactions, very loaded systems where it's enabled.

Michael: Same.

I was, I was coming across customers that had it

Nikolay: Doesn't it depend on the hardware on.

on the CPU.

Michael: Yes.

So I did.

there's like a, there's a tool in progress that lets you check.

Uh, I've forgotten what it's called.

Is it PG test timing?

It's ah,

Nikolay: Something like that.

It's in binary directory, standard package

of

Michael: yeah, I'll find it and link to it.

But yeah, basically my understanding is if you have pretty fast system
system clock, lookups, it, the overhead can be hard to measure, but if

you have slow system clock, then it can be extremely easy to measure.

And I that's the ongress blog post, I think is deliberately
picking a system has a slow system clock in order to

show that it can add hundreds of percent of I overhead.

But when I was looking at on a when I was looking at
it on an OTP workload, very, very simple PG bench OTP

workload, I, I was basically unable to measure it.

I, I got, I think I got a 2% overhead of adding
all of the parameters and it was basically

Nikolay: Y, you know, This is my old idea.

And I, I, couple of times we implemented it in any company.

When we deal with many SGO costs, it would be good when we set up a host
to have a set of micro benchmarks checking uh, Like this scale limits CPU.

We can use this bench for that, or fi for this in the old
life we used Boni plus, plus I remember, and this micro

benchmark checking timing overhead would be also great there.

And like sometimes we, we might have in, in cloud,
sometimes we might have two virtual machines of the.

same class, same type.

And they, but they behave differently.

So it would be good to check it all the time we set up some machine.

Michael: Yep.

Well, yeah, this is your age old.

this is what you are dedicating your professional life to is experiment.

You know, PE get, if you are intrigued as to what it would be
on your system, it's it, it might be different for you for some

reason, maybe for hardware reasons, maybe for workload reasons,
there might be some, some specific way that it's bad for you.

Very difficult to provide general advice and the advice you read online.

Generally be cautious, especially in the Postgres documentation,
then they're gonna be cautious by default because they don't want

to give advice that one person's gonna find horrifically awful.

even if the majority would find

Nikolay: Right.

Back to these three classes of overhead.

I guess the first class is like from UN analyze Inex explain part.

And second is Tracko timing.

Third is buffers let's let's postpone a little bit.

They both are related to this overhead from
how clock clock work with clock is organized.

But the difference is that inside explain, analyze, they are both.

Working, but Tracko timing also working if you haveta
statements, because it's registered there as well.

So regular execution without running, explain regular regular query execution.

We've also included.

So if working with clock is slow Tracko timing
can add some penalty when you use PSTA statements.

Right?

Michael: Same with auto explain.

auto explain runs on every, in there is a, there is a parameter where you

Nikolay: There is sampling and it existed long ago.

I, I, I didn't realize it exists uh, for slow lock.

There is sampling capabilities since POS 13, I guess, but for auto explain it.

you told me, right.

It exists at, for long, like many years already.

It's great.

So you cannot auto explain only like 1% of everything

Michael: If you're to be cautious at first, you, yeah.

You can sample a really small percentage, but naturally yeah, for LTP,
it's probably fine because you're probably running the same queries

over and over, and you don't need loads of examples of them to optimize.

Nikolay: there is also possible observer effect from just logging.

If you, if log writing to logs is slow.

For example, disc is not.

Very fast where you log it.

So it's also can be a problem, but it's slightly different topic.

The third part of buffers, what do you think about other health from buffers

Michael: Well, you were the first to tell me that it's worth
looking into, but I wasn't able to I wasn't able to measure it.

Yeah.

I wasn't able to measure it.

,
Nikolay: there?

That definitely should be difference if you're just run, explain wise.

Many times everything is cashed and then explain
analyze buffers difference should be there.

I I'm sure, but still hundred percent worth having buffers inside.

Explain, analyze as we discussed separately, whole half an hour, right.

Michael: Yes.

previous episode, I actually think it might have, I don't know
if you, if, if this is to do with you, but I think explain Dr.

pe.com deserve some praise.

Cause I noticed today or yesterday that
it now asks for explain, analyze buffers.

So that's quite a

Nikolay: Well, in my opinion, if like when we
analyze a query, we should not do it on production.

As usual, we should do it on a special
environment, which should be a clone of production.

And the best way to get to have a clone
is using database lap engine we develop.

And then there, of course you are in slightly different situation.

Maybe hardware is different.

Maybe you have less memory, for example, different state of Cassius.

And maybe a different fast system as in the case of
database lap engine, because it uses ZFS by default.

And there you, you should focus on buffers.

This is like, should be like our final goal is timing, but inside
it, we focus on buffers inside the process and reducing IO numbers.

Not just buffers, maybe IO numbers like RO rose is also
logical is also important metric to, to keep in mind.

And if you reduce IO, you'll reduce timing.

This is secret of optimization.

Everyone should understand in my opinion, right.

Michael: Yes, couldn't agree more.

and if anybody disagrees we can refer you to episode.

I'm guessing two.

I, it was quite an early one.

Nikolay: Right, right.

Good.

So what else?

we should discuss in terms of starting of working with explain,

Michael: Yeah, well, I, we might be close to time, you
know I wonder if we should save it for another time.

Is there anything else that we have to mention?

Nikolay: well, we didn't discuss particular
notes, like various types of join and, and so on.

Of course, it's, it requires time to learn.

And of course there is documentation.

There are many, I see different people.

Present talk name, explaining, explain.

This is like default default name for such talks.

So not just one person presented it.

So I think all of those talks are useful were checking,

Michael: My, yeah, my favorite is one by Josh.

Burkus I'll make sure to link it up.

He did a really, yeah.

Old, but still I, I listened to it again last
year and it's, it's still perfectly relevant.

there've been some new parameters and sure.

It doesn't cover everything.

Nikolay: New notes, pluralization since then was added,

Michael: Yeah.

But equally.

Nikolay: JIT compilation, which should be disabled on LT.

Michael: Yeah.

I've also done two talks, one at the, be one to try and cover the I
think there's some beginner stuff that he doesn't cover at the beginning.

And there's some more advanced stuff that he doesn't
get to cuz it in an hour you can only do so much.

So I have done two talks trying to cover either side
of that, not doing the explaining, explain part.

so yeah, maybe I'll link those up as well.

Oh.

And also have a glossary

Nikolay: Oh,

Glosser is great.

Yes.

Yes.

So, Yeah.

It's a good thing.

Right?

Good.

So I hope it was helpful for some folks.

let's wrap it up,

Michael: Yeah, I hope so too.

fingers crossed, and also feel free to reach out.

Like, I think this is the kind of topic
that we love and find it very interesting.

I'm Def very, very happy to help people with this kind of thing.

Nikolay: We are asking for topics we can ask right here.

Once again, like we are open, We have a list
of dozens of ideas, but we react to feedback.

If someone asks for a topic, we will prioritize it in, in our list.

Definitely.

And we will try to discuss it soon.

And as, as usual thank you, everyone who is
providing feedback, it's very, very important.

We receive it quite often.

All.

Like at least once per couple of days, it's, great feeling.

I, I would say.

And also please as usual subscribe everywhere you can like everywhere
you can and please share in your social networks and working groups.

Michael: Absolutely.

Thank you so much.

Thanks everyone.

And thanks Nicola.

Nikolay: Thank you.

Bye bye.