Nikolay: Hello, hello, this is
PostgresFM, episode number 101.

My name is Nik Samokhvalov, founder
of Postgres.AI.

I thought about...

Since last time we had episode
number 100, and it was interesting

because we discussed databases
at scale of 100 terabytes and

beyond.

Logically this episode should be
something very simple and beginner

entry level, because it's 101.

You know, in the US, 101 means
something like some introduction

to something, right?

But I guess we will dive quite
deep, I hope we will dive quite

deep into some details of some
things.

Very interesting to me personally.

But also, since a couple of days
ago, Timescale released an addition

to pgvector called pgvectorscale.

I asked developers to join and
they agreed.

So meet Mat Arye.

Hi, Mat.

Thank you

for coming.

Mat: Thank you for having us.

Nikolay: And John Pruitt.

Thank you for coming as well.

It's great to have you.

And Let's start, it's kind of an
interview, but maybe it will

be some informal discussion, because
I don't know, we know each

other for some time and why not,
right?

Let's start from the beginning,
from maybe some distance.

Why do we need at all some vector
search in Postgres and maybe

some vector search in general?

Because recently, not long ago,
LLM became...

It's becoming bigger and bigger
at a very high pace, in terms

of how many dimensions, but also
in terms of context window.

Gemini now supports 1000000, there
are talks about 2 millions.

Although I must admit, Gemini is
terrible in terms of reliability.

We have a lot of credits being
an AI startup, we have a lot of

credits to spend.

And it's a very well-known problem
with Gemini that 500 errors

all the time.

It's making me very sad.

But 1 million context window means
that Probably we don't need

vector search at all and we can
just put everything into 1 prompt,

right?

There was such a crazy idea when
context became really large.

Imagine if we have like 100 million
tokens context window.

You can just put everything to
a single question, or it's an

insane idea?

What do you think?

Mat: SRS- I think there are a few
points to point out.

First of all, there's still the
bandwidth that you need to use

up to send the question to the
LLM, right?

So there is going to be a pure
networking limit at some point.

Another point is that even with
very big context windows or token

windows, yes, you can give the
LLM a lot of information, but

there's actually an open question
about whether it will use all

of the information it gets well.

And there's some academic research
out of LLM that's actually

showing that giving it less information
but more relevant information

in the prompt gets you better results
than giving it all the

information you have, right?

And so, I mean, with LLMs in general,
all of this is purely empirical

research.

There's actually not very good
scientific backing either way,

but with the LLMs that people have
tested, I know that huge prompts

are often not very good in terms
of performance, in terms of

the answers you get.

So in that sense, narrowing prompts
down using either VectorSearch

or TextSearch or HybridSearch,
whatever you want, in many cases

gives you a better answer.

You a better answer.

And apart from that, when this
is a bit controversial, I'm actually

not convinced that RAG is the killer
app for vector databases.

There's a lot of other use cases
for vector databases, starting

from clustering, recommendation
engines, plain old search engines,

right?

A lot of other things that have
nothing to do with the rag and

that increasing context windows
were not, you know, help or help.

These are just orthogonal things.

And the basic thing that what vector
search gives you is semantic

search.

There's a lot of applications for
that.

Nikolay: So not only RAG, right?

So interesting.

Yeah, that's...

Okay, so your answer has 2 points
to summarize.

First, we still need to narrow
down and find...

And if we put everything to 1 question,
it will be not efficient.

I agree with LLM sometimes.

If you put too much information,
the answers are not good.

Quality of answers decrease for
sure.

And second, it's not only about
RAG.

I would like to explore this.

Do you see some good examples beyond
retrieval-augmented generation?

Actually while you think, Let me
just spend maybe half a minute,

because I know our audience is
mostly Postgres engineers, DBAs,

and backend developers.

Of course, many already heard about
RUG, but just a small recap.

It's quite a simple thing.

Actually, you're right.

Maybe vector search is not needed
in RUG.

For example, we can use full text
search there.

So the idea is when some request
is coming to LLM, before passing

this request to LLM, we have intermediate
software which finds

additional information which should
be relevant.

Here vector search can be useful,
but maybe not only vector search,

for example, full text search.

Then We already augment this request
with this information, and

LLM has some context.

Basically, it's similar to if you
go to exam and you have a lot

of information in your pockets,
When you need to answer, you

can get this information out of
your pockets and answer better.

Or use Wikipedia, for example.

Similar.

But you just told me that it's
not only RUG.

What else?

What else for vector search particularly?

Mat: I mean, fundamentally, I think
any time that you are right

now using full text search or text
search, you could substitute

that with semantic search to give
you better results.

Let me just give a simple example
for people.

With the full text search, searching
for a query on car does

not return results that have truck
in the document, right?

Semantic search solves that because
it's kind of, I mean, that

sounds almost magical, but it's
kind of a search on meaning.

So things that are close together
in meaning get retrieved.

So you could use a totally different
lexical structure, totally

different word, and somehow this
type of search figured out that

they are similar in the meaning
space, if you will.

John: Or you could use

it to

augment full-text search
or as a hybrid of the 2.

Nikolay: Have you seen good examples
of this?

Like I saw terrible techniques,
like for example, let's find

100 results from full-text search,
100 results from vector search,

and then combine them.

But always I have a question, what's
the ranking system?

Because it's very different, and
sometimes we want fresh information.

Specific question I would like
to explore maybe later, how to

reflect freshness of data.

We can discuss it maybe later because
I also already want to

discuss what you particular just
released.

But did you see any good examples
of combination of full-text

search and semantic search like
vector search?

Mat: It's hard to say because we
are database providers so we

kind of see the...

We don't see the examples that
our customers have.

We get told by our customers that,
hey, this technique works,

you find it useful, blah, blah,
blah.

But we rarely actually see the
results ourselves.

Actually see the results ourselves.

But like anecdotally, a lot of
people are using hybrid set.

And like, in the AI community,
I think it's almost becoming

the standard.

What is potentially a problem for
Postgres we can go into later,

but That's what we're hearing at
least a lot.

But I will agree that this is totally
all ad hoc, non-scientific,

like, crack my left ear with my
right arm type thing, right?

Nikolay: For me, it looks almost
always like all examples I see,

they look ugly because, for example,
if you think about pagination,

there is no way to have pagination
there.

Although if we go back to vector
search, it's also problematic

there because to go to page number
100, you need to extract all

100 pages, and it's very inefficient
performance-wise.

Let's discuss the project.

It's called pgvectorscale.

It looks like you just append a
suffix scale to pgvector.

It's a very interesting approach.

Let me ask you this.

It's very well known that TimescaleDB
is a very great thing

for time series.

It has a free version, and you
can host it yourself, a community

edition, I think it's called.

I doubt it's true open source because
it doesn't have OSI approved

license.

So it's like some specific thing.

Or maybe Apache license exists,
but it lacks a lot of good stuff.

For example, I think it lacks compression.

Maybe I'm wrong.

Is it?

Mat: Yes, it lacks compression.

So we have an Apache version, we
have a community version.

Nikolay: Right.

I think you can use the extended version,
still not paying if you

host it yourself, but it cannot be
considered true open source because

this license is already something
like hybrid and so on.

Mat: We call it source available.

Nikolay: Yeah, yeah, this is a
common term.

So then Postgres itself has a very,
very, very simple and permissive

license.

That's why many commercial products
developed on top of it.

Then we have pgvector, which is
a separate extension, which in

my opinion should at some point
go to core Postgres.

But there is a big question, how
come?

Because it's breaking all the rules,
because just creating an index

you might have results which differ
from original results without

index, right?

It's something which we didn't
have before, because as always,

indexes just sped up queries.

Now we can change results because
it's approximate neighbor search.

This project, I think it's...

Which license is it?

I don't remember, but I think...

Yeah.

It's also Postgres license, Right.

And maybe that's why you also developed
your product on top of

it using Postgres license.

But it's very interesting because
if before Timescale had this

major product, everyone loved,
compression especially, but only

if you go that route, source available,
or you go to Timescale

Cloud, now you release this thing
on Postgres license, meaning

that anyone, any cloud provider,
most of them already have pgvector,

so naturally they should already
be considering adding your extra

to their services.

I think, of course, you've thought
about it very well.

So my question is why you changed
your approach compared to Timescale?

Mat: Yeah, so I'll speak for myself,
maybe not for the company.

I think, look, we all love Postgres,
but I think Postgres needs

different things in different areas.

Postgres, in the time series area,
Postgres already has a lot

of functionality to build in and
we are already building upon

a well-known foundation where I
would argue that given our broad

definition of time series, in many
ways Postgres already dominates

the market.

So we felt that we could build
a company while contributing some

things to PostGrid, but also quite
frankly making a business

for ourselves and making money
in this area by preventing hyperscalers

from making all the money off of
our work, which is what the

TSL license does.

It doesn't stop any individual
from benefiting from our work

or really any company, it only
stops the cloud providers from

making money off our work.

And that was then for pragmatic
reasons so that we could make

a business.

In terms of vector, I think the
whole vector market is much more

nascent and much younger.

I don't think Postgres has won
the market a lot.

You can look at how many other
databases have gotten millions

of dollars in the past year to
develop their own solution.

This is a very hot, fast-moving
market where I don't think Postgres

yet has a particular...

It hasn't won, right?

And so, because we like Postgres
so much, we wanted

Nikolay: to know when

Mat: Postgres wins.

And so...

Nikolay: There are many benchmarks
where Postgres has very poor

results with pgvector on ivfflat
index without HNSW and so on,

right?

And they're still around.

Mat: I will say with pgvector 0.7,
things have improved on the

pgvector side quite a lot as well.

But yeah, and I think not only
in terms of benchmarks, I think

in terms of actual usage, if you
know, last episode you talked

about 100 terabyte databases, We
want to reach 100 terabyte databases

that have extra data.

We have to do a lot of work to
make that happen, right?

And I think that's the ultimate
goal.

And there should be the ultimate
goal of everybody in the community.

We thought that it was more wise
to help the entire community

succeed at this point in time.

Nikolay: Yeah, when I saw benchmarks,
like 100,000, 500,000 vectors,

I was like, what's happening here?

People say, oh, it's already a
lot.

It's not a lot.

You take any company, take their
databases, and they have already

so many data entries in their databases.

Out of those entries we can create
vectors usually.

It means that we should speak about
billions already, but it's

not there yet because it's problematic
because vectors are large

and building indexes is slow, takes
a lot of time, and the latencies

of search, of course.

Everything is quite slow.

I'm glad you did your benchmarks
starting with 50 million vectors,

already very good.

I'm super glad to hear that you
talk about terabytes of data.

By the way, some people say, okay,
for us, for Postgres guys,

not for Timescale, regular Postgres
guys, 1 terabyte is usually

1000000000 rows.

With Timescale compression, it's
many more.

1 terabyte should be maybe tens
of billions rows, right?

And I know...

Tens of billions.

Right, right.

So it's a different order of magnitude.

But if we talk about pgvector,
50 million vectors.

Let's talk about...

This project has 2 things to bring
on the table.

Maybe let's close first with a
non-technical question, because

I cannot skip this question just
sitting in my head.

So pgvector had ivfflat index originally,
then HNSW was added.

I remember NUAN participated, starting
with a separate project,

then decided to contribute to pgvector.

So now it has 2 indexes.

And pgvectorscale brings a third
type of index, right?

It could be a pull request to pgvector,
no?

Mat: So we actually talked with
Andrew Kane about this.

The issue is that we are written
in Rust, not C.

And Andrew thought that it would
be better as a separate project.

I can't say I disagree.

I think either could have worked,
but I think given the language

difference, it makes some sense
to have it as a separate project.

But we did all for...

Nikolay: Yeah, well, makes sense.

Makes total sense.

But why Rust?

Mat: Because I like to work quickly.

And I thought Rust would allow
me to do that more than other

things.

I should say I'm 1 of the people
that worked compression Timescale

and we did, you know, our compression
system had our own data

type on disk and I remember the
pain of having to make sure that

the way you are writing data on
disk would work very well.

And the cross-over platforms, big-end,
little-end, all of these,

the alignment issues, it's just
a lot of double checking that

you have to do to make sure everything
is correct.

I wanted to avoid this.

And with Rust, a lot of this
could be a lot simpler.

So, like...

Nikolay: Yeah, makes total sense.

Yeah.

Mm-hmm.

Right, right.

So, this makes total sense, but
for end user, it's interesting.

Especially those who self-manage.

So you need to take Postgres, then
you need to bring pgvector,

then you need to bring pgvectorscale.

It's an interesting situation.

We do

Mat: have web-based packages.

Nikolay: This is interesting, but
now I understand better.

Let's talk about the index itself.

What can you tell us about the
index compared to HNSW?

I know HNSW is like a memory thing,
this works with disk, so

this is definitely the right thing
to do if we talk about terabytes

of data and billions of rows and
so on?

Mat: Yeah.

So I'm going to hand wave a lot
here because this is a lot of

history, but it turns out for various
reasons that the best way

to index vector data is using a
graph structure, so a graph database.

You can imagine each vector is
a node and is connected to other

nodes that are mostly that are
close to it, right?

And then there are some for edges.

And the traditional problem in
these types of indexes is getting

from 1 end of the graph to another
end of the graph.

So your starting point is very
far from your query vector, which

is what you're looking for.

It takes a lot of hops in these
graphs.

And HNSW and DiskANN are pretty
much different ways of solving

that basic problem.

HNSW solves this by introducing
layers into the graph, where

the top layer where you start only
has long distance edges.

So it kind of allows you to jump
a long distance but close to

where your query is, and then you
go down a level in order to

make more fine-grained jumps.

And there are several levels.

By the time you get to the lowest
level, you are kind of in the

most fine-grained area of the graph.

And those top levels are the things
that help you solve this

long jump issue, if you would.

This KNN doesn't use multiple graphs.

Instead it kind of uses a neat
trick in the construction to...

How do I put this?

It puts in enough long edges to
be useful.

So the way it constructs the graph
is modified from the regular

way these types of graphs are usually
constructed, specifically

to inject these long edges into
the graph, which probabilistically

allows you to jump faster to where
you need to go.

And this kind of flat structure
is what allows you to keep a

better locality of where you are.

So instead of using multiple levels,
the flat structure allows

you to make fewer hops in memory,
which allows us to work on disk,

on SSD rather than RAM.

Because...

Nikolay: Yeah.

I noticed, by the way, maybe it's
like a side note, but I noticed

you in Postgresstab you use local
SSD disks, right?

Not like EBS volumes.

Is it fair?

I mean, they're ephemeral.

If you restart the machine, you
lose them.

So in your benchmark particularly,
I noticed this small point.

Mat: Our benchmarks very concretely
benchmark PyCon against self-hosted

Postgres.

And the most self-hosted Postgres
people I know actually use

NVMes and they do like...

Nikolay: But not local NVMes.

Mat: But you can do backups other
ways, right?

You could

Nikolay: do string

Mat: replication, you could do
other ways to get around this

issue of things don't go away.

Nikolay: I'm just very curious
how much it helped because of

course latency is better, discrepancy
is better.

Mat: I can tell you it helped immensely
and I can say that we

are right now actively thinking
about how to bring this kind

of performance to cloud as well.

Stay tuned, I think we'll have
some exciting stuff coming out.

But yes, that's a very astute observation
because, you know,

Quite frankly, none of this works
on EBS.

You need the random read performance
of SSDs to make this work.

So by the way, I would be shocked
if Pinecone wasn't using...

Nikolay: ...VMWare's NVMes?

Mat: I don't know.

I have no idea what they use, but
I would be shocked if this

is EBR based.

Nikolay: And for these 50 million
vectors, how big was the database?

Like in terms of...

John: The disk and index was about
40 gigs.

The entire table, I think, was
around 250, somewhere between

250 and 300 gigs.

Nikolay: Right, and the machine
has definitely less than that,

so of course it was not cached.

I mean, we do need the disk.

Okay, and back to indexes.

DiskANN, and I saw you mentioned
in the article and the project

itself, Microsoft work, but you
modified this work, right?

But the question is, like, HNSW
and this DiskANN, both are approximate

nearest neighbors algorithms.

Can we say one is better than another
for most of workloads and

data types, data sets?

Or there are cases where one can
win, there are cases where another

can win.

That's what I think.

John: ANDREW BROGDON-PICKETT That's
windows index supports concurrent

index builds, which could make
it faster, at least for building

the index.

SIAMAK

Nikolay: ROSHANI For building the
index, So, and yeah.

But for search?

Mat: For search and accuracy, look,
we haven't benchmarked everything.

The thing is that we have benchmarked,
we've seen higher throughput

and higher accuracy.

The trade-off was always kind of
better than HNSW, but you know,

we haven't benchmarked everything.

We've kind of concentrated on modern
embedding systems.

So, for example, we haven't gotten
to benchmark very low dimensional

vectors like 128 or 256, the lowest
thing we've benchmarked is

768.

So a lot of caveats and I think,
you know, the space is so new

people should actually test on
their own data, but I can say

in terms of research we haven't
seen where we're worse yet.

Nikolay: People should always test
on their own data, even if

it's old school full-text search
and so on, because who knows

what you have, right?

Mat: Absolutely.

John: That's one thing that made
benchmarking so hard, especially

versus our competitors, because
a lot of the specialized vector

databases have very few parameters
that let you actually control

the index, whereas on Postgres
with both pgvector and pgvectorscale,

there are several different parameters
to tune the build of the

index, and then also query time,
plus all of the Postgres settings.

And if you're self-hosting, you've
also got OS and machine level

things to play with.

So there's just a mind-boggling
number of variables you could

play with if you had the time and
money to spend on it.

Nikolay: Yeah, and in pgvectorscale
particularly, I also checked

the source code and documentation
that we have so far, and I

saw also parameters you can touch.

A couple of them, like query time,
and if you can be also adjusted

during build time.

What can you say about some list
sizes and so on?

Could you search list size?

And during build time, num neighbors,
how many neighbors?

Do you recommend trying to tune
this so far now?

Mat: So we tried to put in the
small defaults.

That is what we saw as the best
combination.

But obviously, we wanted to let
people also experiment.

But in terms of build parameters,
I would recommend keeping them

as a default.

The runtime parameters is really
where I think people should

experiment because that allows
you, at query time, to make the

trade-off between accuracy and
speed, right?

And there's the re-school parameter
that I think is the 1 we

recommend to actually tune.

I just realized today we probably
should change the default is

pretty low.

That was my fault.

But that is the parameter I think
people should play with.

John: Yeah, that 1 seemed to have
the most impact on both speed

and accuracy.

At query time.

Nikolay: Right.

Okay, so 1 more question about
the index.

You call it StreamingDiskANN,
right?

Why streaming?

How does it differ from the original
Microsoft implementation?

Yeah,

Mat: so I think both the HNSW implementation
and the original

DiskANN implementation, you tell
it ahead of time how many things

you want returned.

And that just doesn't play very
well with Postgres, quite frankly,

because the Postgres

John: model

Mat: For indexes, Postgres, the
other part of Postgres, can always

ask the index for the next tuple.

And this is done so that you can
filter your results after index

retrieval.

So for example, let's say I want
to find the closest vectors

to a given query that also meets
some other criteria that belong

to the business department or the
engineering department, right?

The vector index only fetches the
closest things to the query,

and then what Postgres does is
that after index retrieval it

will filter out department equals
engineering, right?

Now let's say your closest hundreds
of vectors are all from the

business department.

That means if that-

Nikolay: It will

Mat: be fast.

What?

Nikolay: That means- I mean, in
this case, it will be fast because-

Mat: Right, but if your parameter
is, you say you're returning

the closest 50 from the index,
then no results will be returned

at all.

Because the index returns 50 results,
Those 50 results are then

filtered by department equals engineering.

Nikolay: None of them bad.

Different department, right, right,
right.

A lot of work to be done, but 0
results,

Mat: right?

Exactly.

And there are 0 results because
there's this arbitrary limit

that you give at the beginning
of the query to tell you, hey,

retrieve this many results.

Whereas in reality, Postgres has
no idea how many results you

need to retrieve in order to match
both the query and the department

equals engineering, right?

So What the streaming part of the
algorithm does is it removes

that restriction and makes the
algorithm work in the way Postgres

expects other indexes to work.

So you can tell the index, hey,
give me the next closest thing,

the next closest.

Actually you could traverse the
entire graph, your entire table

like that.

And that makes all of your queries
that have a secondary filter

completely accurate.

Nikolay: Sven And this secondary
filter, it's a different column,

for example, text or integer, right?

Or not?

Mat: Rony Yeah, it's a different
column.

Often it's JSONB, right?

So you could have an article, you
could have a list of associated

tags with it in JSON-B, and you
can say, hey, find me all the

articles about cars that also come
from country USA, or any kind

of other metadata.

So it's this combination of semantic
and metadata, which is actually

incredibly common.

Nikolay: Right.

But there we have usually either
a situation when we have Btree

and gin index and Postgres needs
to decide which 1 to use and

then still apply a filter, or we
need to use something like additional

extension like GiST, I'm
bad with names as usual, and then

try to achieve single index scan.

But here it's not possible.

I mean, ideal world is single index
scan without additional filtering.

Mat: Yes, that is the ideal world.

Right now, none of the indexes
supports that.

And so all of the best you can
do is the best, the state of the

art on Postgres right now, and
that would over you, state of

the art period, but we could leave
that argument for another

time, is you can hope that you
can retrieve from the vector index

and then post filter and still
have accurate results.

Nikolay: I can imagine, for example,
if we take this department

name, put it inside this input
text which builds a vector, and

then additionally we filter by
integer like department ID, it

probably will work better in this
case, right?

Because first we apply semantic
search involving in the query

which department we want, But then
we think, okay, this filter

will make final polishing.

But it sounds again like some ugly
solution.

Mat: So...

Yeah, it's not only ugly.

I'm actually not sure, it won't
work, but like using my intuition,

like the department engineering
in the text would kind of skew

the semantics away from the actual
thing in the text you're talking

about.

And so I'm not sure that would
combine well in this kind of semantic

multi-dimensional space.

John: You could maybe
add a dimension and synthetically

set a value in the dimension to
represent which department it

is.

Nikolay: And 1 more dimension for
time, like timestamp.

This is so natural.

I'm very curious if there are some
works already in this direction,

because everyone needs creation
time, like publication time or

something for each data entry.

I didn't see good discussions about
that yet.

Because if you, for example, we
loaded almost 1000000 entries

from Postgres mailing list archives.

Then it started working great,
but when you see some discussion

from Bruce Momjan from 2002 about
something, I don't know, I

remember just some cases, It's
already not relevant at all.

You think, maybe I should delete
all data, but still, it might

be relevant.

What we did, ugly solution, we
return 1,000 or maybe 5,000 results,

and then we dynamically apply some
score, I think it's logarithm

approach for time, so we add some
penalty if the article, well,

email is very old.

It quickly becomes less relevant.

We just combine it with score,
pgvector provides us similarity

or distance.

This works well, but sometimes
you have 1 second latency on 1000000

rows dataset, and this is terrible
and doesn't scale.

So this problem of additional dimensions,
I think it's huge.

We need to extend original vector
with non-semantic dimensions

and use them in filtering and achieve
1 single index scan.

Good performance, right?

Mat: I would say that in the academic
literature or there is

some progress being made.

That is, the filtering, this can
end paper, which I believe is

2 years old, which is ancient in
this space.

There was another paper from a
group out in Berkeley talking

about building and filtering into
these graph-based indexes as

well.

But you gotta walk a crore people
you can walk.

We just haven't had any chance
to really implement these inside

this index, but there is, there's
kind of a chaotic work in this

area.

Nikolay: Right, for this algorithm.

John: You'll be able to signal
that like certain dimensions,

if that's where you're putting
your filters, are, that they would

need to be exact versus approximate.

Nikolay: Right.

And Do you think this KNN approach
is better than HNSW in this

particular problem?

Mat: For filtering, yes, I do.

Obviously, I'm biased, but I think
the simplicity of going from

multi-levels to a single level
really helps in a lot of these

things, just because there's a
lot less edge cases to consider.

Nikolay: Let's not forget...

John: This helps as well, right?

Mat: Yeah, I think Streaming was
a lot easier to implement because

it's a single level.

Nikolay: Interesting.

Let's not forget to talk about
compression, because I'm sure

we need it in vector world, in
vector search.

We need it.

I saw in a recent pgvector there
are ideas, let's not use integers,

let's use floats and so on, with
a kind of compressions, so to

speak.

But you talk about real compression,
right?

Maybe some experience from TimescaleDB
extension, or no, or it's

different.

Because there is time series, I
remember articles, TimescaleBlock

has excellent articles, but maybe
it's different here, right?

Mat: I don't think I directly used
any of the algorithms from

the time series space for this,
but the basic insight of using

the statistical properties of the
vectors you have to kind of

make compression, give you better
results is exactly what led

to...

So we have an algorithm called
Statistical Binary Quantization,

SBQ, And it takes quite a simple
but well-known algorithm called

BQ and kind of adapts it to your
dataset in a better way.

You know, we have a blog post about
it.

It's pretty simple.

It's pretty much using the means
of each dimension and the kind

of standard deviations to kind
of better segment your space,

if you would.

And yeah, and we just, honestly,
we took a very experimental

approach.

We took various datasets and we
tried different things and this

turned out that it worked.

It added a few percentage points
to the accuracy, which once

you get into the 90s, a few percentage
points is a pretty big

deal.

And so, yeah, it's, you get to
read about it in our blog post.

The algorithm is fully explained
there.

Nikolay: To make sure...

I'm just noticing, and...

John: The compression that Mat's
talking about is happening

in the index and the TimescaleDB
compression is, you know, converting

row-based into columnar and then
compressing each column in the

heap.

We actually tried compressing the
vectors in the heap with TimescaleDB

compression. And seemingly random
vector strings of numbers don't

compress very well.

So that didn't help very much.

Nikolay: RADUKIJ YURIYENKIN Because
in TimescaleDB, one of the

key ideas is that for time series,
it's like values are changing,

not jumping, how to say.

So there are deltas and these deltas
are quite low and so on.

This is what I remember from those
blog posts.

For vectors this doesn't work,
I understand.

For index compression, how much
could we expect in terms of compression

ratio to achieve?

Mat: Uses 1 bit per dimension.

The uncompressed version is float4.

It's a very easy calculation.

It's always a 32x compression ratio.

Nikolay: Roman Karpukhin Good.

Understood.

Great.

So yeah, anything else in technical
area we should mention?

Mat: James Johnson

Nikolay: That helps

John: with the size, but it also
helps with performance because

you can fit

Nikolay: more in.

Fewer buffers to load to the buffer
pool.

This makes sense.

I remember your blog post also
mentions storage costs you compare

with Pinecone in terms of how much
do you need to spend each

month to store 50 million vectors.

The difference is very noticeable,
I would say.

So yeah, this makes sense for sure.

Anything else, like technical stuff?

Anything maybe you're working on
right now?

Mat: I think about PG vector scale.

That's about it.

We haven't talked about PGAI, which
is the other thing we announced

this week.

Nikolay: This is a Python untrusted,
so it won't be possible

to run it on managed services,
as I understand, but it allows

you to...

Or it's something different?

Because it makes calls to OpenAI
API or...

It

Mat: does.

So, with untrusted languages, you
can't allow users on clouds

to write their own functions.

But if the functions are included
inside an extension, that's

fine.

And it's easy to see because most
extensions are written C, which

is completely untrusted, right?

So the entire point of the PGA
extension is specifically so that

this could be run on the clouds.

Nikolay: So it limits capabilities,
and if a cloud vendor decides,

verifies everything works well,
nothing, Bad calls cannot be

done through it.

So we are in the

John: center.

It's almost like whitelisting certain
programs.

Nikolay: Interesting.

Makes sense.

So the idea is to make it really
simple to create vectors from

regular data types, just transparently
calling these, just with

SQL, right?

I think there are other similar
implementations of this, But

I like your idea of betting on
cloud providers, including this,

and even not providing untrusted
languages, capabilities.

It's interesting.

Mat: Yeah, there are a few things
that do similar things, and

I was always curious why they didn't
just run the Python code

to do this.

And so that's what we did.

A lot of people do very complicated
stuff to get the same result.

I don't know why, but yeah.

We had to work that.

Nikolay: I like this approach myself
very well.

I mean, I do it a lot for many
years, just select and call something

externally.

But of course, there should be
a huge warning on site.

It doesn't scale well because if
Python calls...

You basically add latency to your
queries of the primary.

Primary CPU is the most expensive
resource you have.

If you want to generate a vector,
you cannot run it on a replica

because you need to write it.

You must do it on the primary.

While this query is running, it's
a primary node, and offloading

this work on Python application
nodes makes total sense because

the database doesn't notice this
at all, the primary doesn't

notice.

You only speak to it when the result
is already retrieved from

OpenAI or another LLM provider.

This is handy, but very dangerous
in terms of scalability and

performance and future issues.

Mat: Completely agreed with you.

This is a way for people to get
up and running quickly and testing

and experimenting.

And we do have another project
called pgvectorizer for when you

need to batch things up and scale
it and do it in the background.

We have all of that as well.

Nikolay: I haven't seen that, that's
interesting also

Mat: to check.

Nikolay: Yeah,

John: but all of that plumbing
that you have to write to drag

data out of the database and then
send it off to an API and put

it back in the database.

You end up writing so much code
that when it's in the database,

you don't have to do not to mention
all of the bandwidth you're

consuming and latency there.

Nikolay: Right.

I remember I was copying images
from S3 in SQL.

And then, of course, it's similar.

You call some APIs from Postgres.

It's great, but it's an interesting
direction.

With warning, for start it's good.

I think I don't have any more questions.

I'm looking forward to trying this
project.

We wanted to do it before this
call, like yesterday already,

but we had some technical difficulties.

I hope we will solve them soon
and try pgvectorscale.

For our case, I will tell you when
we have results.

Thank you.

So as a summary.

Mat: There's 1 more thing about
that.

This is a new project.

We're trying to be very responsive.

So if you run into any problems
at all, you know, file GitHub

issues or contact us on Discord.

We are more than happy and very
eager to talk to anybody.

This is a young project, as I said,
so all feedback is very young.

Nikolay: Sure, yeah, that's a good
point.

Honestly, seeing this work in Timescale
company repository on

GitHub, it makes me have some good
expectations in terms of if

some problems are encountered that
will be addressed and so on.

As a summary, let's say everyone
who works with vectors in Postgres

should check out this new type
of index and compare.

This is great.

I think there are many people who
do this right now and looking

forward to results in our case
and other cases as well.

Thank you for a good project

Mat: and very interesting.

Nikolay: Thank you.

Good luck with it.