026 Postgres year in review 2022
===

Michael: [00:00:00] Hello, and welcome to Postgres fm, a weekly show about all things PostgreSQL. I'm Michael, founder of PG Mustard, and this is my co-host Nikola, founder of Postgres ai. Hey, Nikolai, what are we talking about this week?

Nikolay: Hi Michael. This is episode number 26. And that means since we didn't, miss any week, and since like each year has roughly 52 weeks, it means that, uh, it's exactly half a year doing podcast every week. How good is.

Michael: Yeah. Um.

Nikolay: It's pretty good, right? Like I, and when we, when we started, of course, my goal was not to miss any weeks, like, because when you start missing weeks, then you start missing two weeks. And like, it's kind of, but we are, we have good, steady progress in, in, in this term, right? So we started in June, I guess, or, or in July.

Michael: I think it was the end of, was it beginning of July? I'm not a hundred percent sure. Well, congratulations to you. Thank [00:01:00] you. And thanks to everybody who has been giving us encouragement and feedback to keep going.

Nikolay: Yeah. This is our fuel. Definitely. and we, we receive it every week, which is very encourag. . So, this was, just some side note, but, uh, since it's uh, the end of the year, let's maybe wrap up some summary of the year in terms of pogo ecosystem and broader community it'll be probably very like opinionated list. Uh, not ideal. We don't have the goal to have ideal list, and it's just something which, for example, I remember and consider important, interesting, entertaining and so on. So just how, how we name it. 2020. most interesting, most interesting, uh, facts and events. Right? Okay. I have a list of five items. You have also a, a few items to add. Let's start from my list because it's my turn to define the topic [00:02:00] and uh, I will start from the observation that. Ecosystem has very strong startups and despite of, issues with the global, uh, in, I mean fundraising and so on, this year showed. Very, very good results in terms of money waste by bogus related startups and also new startups appeared. And I would, , mention, I, I would, I would start from Ivan here. They are not strictly like pogs, only think, but. Of course, but, they started, as I know from, delivering services, first of all, Pogo and Kafka, then extended and so on. So they are quite strong in pogo and they, previously, in the, in late 2021, they, achieved 2 billion evolution and total raise this year achieved 420 million. It's quite, quite impressive. And in February, two more companies, reported that, they raised more [00:03:00] and, they had new rounds and they achieved $1 billion evolution. these two companies were, were, has and time. If I'm not mistaken. Right.

Michael: Yep.

Nikolay: And uh, also we have a bunch of other companies who also raised the lot. For example, super base, and we have new startups. very recent one is, uh, HIRA, right? Hydra. How, how to, how to pronounce it,

Michael: I say the latter. I say hydro.

Nikolay: Mera because I.

Michael: the, yep.

Nikolay: and they, and interesting that, also Neon of course, had released this year. They didn't appear this year, but they had released this year. And, uh, we will talk about Neon in in my different title also a little bit. And, uh, I like how. of these, uh, three, three startups, uh, has super base, neon and hydra. They choose their style of value proposition. We are this, but open source, right? So super base is open [00:04:00] source, Firebase alternative, uh, neon is, uh, open source, uh, of rural alternative. And Hydra is the open source, uh, snowflake. and all of them are built based on pos and this is super great and this, I think like in my opinion, these. Item in my list. This is most strongest, biggest and most interesting because, these humans, um, bring a lot of in energy, not only money, but a lot of mines and, uh, a lot of new small projects, open source projects, and so on and so on. This is great. This shows how pogs community grows and POGS ecosystem grows. and, uh, this is new heights for pogs Ecosystem. But it also, uh, I think in my personal opinion, this also, uh, means that pogs community is much bigger than pogs project itself because, uh, they are different companies, different, uh, organizations. [00:05:00] Uh, different people and, uh, pogs is just in the center, right? But, but there are many things around and they are growing in terms of business, in terms of, uh, people, user, user base, and so on. What do you think?

Michael: Yeah, it's really cool. it's an exciting time to be in the Postgres space for sure. A lot. Uh, something I, uh, don't think I realized coming in that a lot of this money would be around, a lot of them are hosting providers, which is super interesting. That seems to be where a lot of the. , um, or at least that's how they're monetizing. It's, it's kind of, that's, it's the cloud offering that they're charging money for, hence they can be, uh, open source in a lot of the cases. but it's really cool for Postgres core as well. I think a lot of them are hiring maintainers or trying to hire people to work on Postgres core. I think it's really healthy for it to be, for, for that group to be gradually, um, populated by people from different backgrounds. I. In the past, it's been a mixture of people in the community, [00:06:00] but a lot of consultancies and a lot of industry, but not so much, uh, techno, you know, actual posting providers contributing back to Postgres and I, I think this year with the companies you mentioned, but also even with the likes of Amazon hiring Postgres. Core team members, it's, I think it's an exciting time and, and hopefully, uh, that group will be, be able to achieve more with more resources behind them with more money involved.

Nikolay: Yeah, it's good that you mentioned, I, I think this is just the model that works. Today, uh, to provide, um, cloud offering like managed pos, but with some additions or very modified POCUS version. But, uh, it's just a model that works best today. And, uh, indeed, uh, like RDS is, is I think still, of course, obvious leader here, but others try. S be somehow different to add something and so on. And, they are interesting [00:07:00] in, in various senses. Not, not all of them are just hosting, like if you talk about supervisor timescale or has, or they, they, like, they're, they're not just positives, right? they provide positives plus something like time, time serious, uh, extension. It's not only like it's extension, but it's, it feels like a different database, maybe time, mean timescale or Azure and Super waste. They provide API or, or, or graph scale. Out, like immediately you have you, you speeding up development. , that's, that's interesting. But indeed, yes. Uh, the model works, mo model works be best. Now it's, uh, cloud offering. So, so we have a lot, a lot, a lot of options to choose from. If you want just positive database plus something, maybe you, it's a headache right now already. So dozens of options to choose from and it's, it's, you need to compare and. [00:08:00] It'll take time, but it, it's, it's good. I think, uh, it, it, it also, also emphasizes my idea that POGS is not just single something. It's very, big and, uh, consisting on very different parts of. Po I mean, community ecosystem. It consists of many different things. And, uh, I see this as a hu huge bazaar. It's indeed open source. Like if you, recall this, bazaar versus cathedral from Richard Stallman, right? So Poss is indeed Huge bazaar and, and we have a lot of players and it's, it's great. And also wanted to mention, if you mention RRGs, for example, uh, I also wanted to mention that, this year I changed my mind about, uh, Google Cloud sql, because previously they had only like eight knobs and it was, it was feeling, my feeling about the. Their offering of POGS was, it's quite weak. I cannot, uh, run serious project [00:09:00] this year. Complet, I completely changed my mind. maybe because Hannah Crossing, visited our POGS TV episode. Like it was great episode about vacuum. Can, can't, uh, uh, recommend enough it, but uh, also like it's obviously it's improving like in terms of product and, and, uh, also interesting. Okay. This was number one, uh, item on my list. Second item is, um, uh, actually our podcast started this year, right? So, so this, this is super important news for Postgres community under a system. and uh, and also positive TV started to become more active as well. So we, first of all, those who don't maybe know, we also publish our episodes on, uh, YouTube. Uh, . Sometimes it's more coming and sometimes less depends. But, uh, besides, uh, our wonderful, uh, podcast episodes, Positive is a YouTube channel. It exists for, for a few years, but this year, it started to be much more active [00:10:00] We, uh, invite various guests. Sometimes it's interview, sometimes it's, uh, to redo the talk. to record it and distribute it. It's called Pogo Open. Talks to provide good talks to wider audience. also we have a good collection of playlists, like for example, POCUS backups or POGS replication and so on and so on. And, uh, those are materials from. Other channels, and I think it's already approaching 500 videos about pogs. So, so go check out pogs.tv. It'll reject you to YouTube and you'll see a lot of interesting stuff, always to something to learn about. I, I actually, when I invite some guests, I learn a lot. This is also, I do it also like it's also a selfish. It's not only to for community reasons to share, but also like I, okay, I have like one hour and I can dive into some topic as deep as possible because the guest, who is working in this area is maybe one of the [00:11:00] best, experts in this area in the world, right? So it's really great. So TV FM and maybe other, other source also, they also publish materials. These days, maybe Covid h helped under realize that, uh, online distribution, online events, they also matter because not everyone can come to, to conference offline. and also you can, you can stop, listening. pause and then return next day if, if you need to interrupt. Right. So it's also much more convenient to consume information in such way. Of course, offline events, they give, uh, benefits from, um, live communication, but online events are also good and, uh, recordings also. Good. So this is, this is, um, num item number two. What, what do you think about it?

Michael: Yeah. Um, really, I like it a lot. I think there's been, not just us, I think there's been a lot of people providing a lot of good Postgres content this year. and I wanted to give, I think the Postgres TV thing is, is worth a [00:12:00] special shout out that you do quite a few of the sessions live. So if people, they're quite, they're on quite advanced topics, but if people have question. If people want to be able to ask questions of those experts as well, they can join live and ask you them in the chat and you'll, you'll pass those on So that, those are quite a unique opportunity I think that you often don't get unless you can go in person to a conference. So that's really great. Couple of others that I wanted to give a shout out to were to Tobias Petri, who's doing more beginner friendly tips on Twitter and his website sequel for devs. that's really gained a lot of momentum this year. He's doing great. And Hussein, nasa, who has a really popular YouTube channel for backend developers. Uh, I'll link that up as well. I think he does fantastic work and explains concepts really simply. And then, , also wanted to give a shout out to the Postgres Weekly newsletter. I think still every week. I know, I know it's been going quite a few years now, but every week they, provide a really good newsletter for all, all kinds of [00:13:00] topics around Postgres.

Nikolay: Yep. Agree. Uh, okay, item number three, uh, shouting in. So, In the beginning of the year, I had strong impression, like, we definitely need charting. And MyQ has vites. A lot of big companies who use my sql, they use vites. PSUs likes it, and it was like we have cys, I mean, POS ecosystem has cys, but it's only partially available because, Open Source Community edition. It's, doesn't provide, for example, online retarding. It was before. I mean, I, I'm, I'm discussing what was in the beginning of the year, but this year changed it. First of all, CTOs published everything as open source like mi, Microsoft published because it's part of Microsoft now. And this is good news. So it's a fully available as open source. Great. And couple of young projects, uh, in this area. , started First is, S P Q R.[00:14:00] guys who I know, the Russian speaking guys. And, uh, with very good experience. They try to build very simple, yet powerful shouting system with exists on GitHub. And then, uh, also some guy from San Francisco, I think I, I don't remember the name, sorry. Uh, this system called PG Cat. subscribed. I'm subscribed to both. Projects and they are very active in terms of development. So interesting to check them. S p q is in go and ette is, is rust and they are already almost every day something. This is, this is super great to see. and PG cat actually originally the idea of shouting there was ette was created as I understand, uh, as some proxy, like replacement, forage bounce, or probably connection pool, but shouting those there originally only like. some, um, very simple explicit charting. You say, I, I need [00:15:00] to route all that. No, that's it. But as I understand later, it became more complex and com comprehensive and so on. So now it's already something interesting to if you need charting, but of course, uh, only. Projects need starting because, uh, many companies showed you can grow to billion dollar evaluation, having SaaS or eCommerce and still have one or, or few monolith, big positive databases, uh, or split to services, big services. But, At some point, any company, if it needs to grow a lot, it'll need shing. Some service will need to be sharded. So Shing is needed. It's just needed only to few users, but it's still needed. And we have some progresses here. It's not perfect, of course, like there is no obvious, default solution answering all questions. No, but there is very good promising movement this year. Cys Plus. [00:16:00] Plus these two, two projects,

Michael: Yeah, I think PG Cat sounds particularly exciting and really cool that it's the idea of putting it in the PO in a pooler with, I think it has also load balancing and failover support as well. That seems really smart to me, that you could put it in place and benefit from those features before you need charting, and then make use of charting later if even when you need it. that sounds very, uh, sensible to me.

Nikolay: right. And this middleware should be very light in my opinion, if you, if you aim to work in OTP context, because if you use for example, pocus there, it's also possible, but, it's quite heavy. and it adds latency overhead. So, so of course, uh, war testing each, each particular is, uh, case, but, having some pro and VI test, my sequel VI test also has proxy alert.

It's called V Gate and it, and it's good if it's very light. The problem is, uh, how to support [00:17:00] all, all Posy in this case, because when you start from zero, posy is quite, uh, rich.

Michael: Yeah. A couple more things I wanted to mention on this front. Actually, there was a really good notion po uh, blog post. I'm not quite sure if it was the beginning of this year or a little bit earlier, but they implemented, I believe, , people refer to it as application side charting. not a great acronym,

Nikolay: uh, yeah, yeah, yeah, yeah. Well, it's, it's, uh, I, I, I promote this approach, uh, this, uh, acron, it's application level charting is less, uh, less offensive, right.

Michael: but yeah, the, the VI stuff's super interesting and. I think this, for me is actually a real threat to Postgres. Not necessarily on a technical level, but on a marketing level. I'm, I'm seeing, I'm getting kind of flashbacks to the Mongo DB days where people would be offered kind of web scale, you know, uh, you are only a startup, but you'd never have to worry [00:18:00] about skating in the future. And suddenly we've actually got,

Nikolay: And it doesn't matter. It doesn't matter that one node behaves, uh, 10, 10 times worse than one node in politic. It doesn't matter,

Michael: Yeah. So, but people don't take that into account when they're starting start, when they're starting up. And actually they don't notice because the performance, when they have very few users is, is fine. You know, and it's, and it is fine. The, the solution is fine, but I'm seeing startups that I, that last year I would've expected to be on Postgres picking. For Tess and MySQL because of Planet Scale have have really nailed their marketing. And I think also I've got some really good developer friendly features, but mostly I think it's that same marketing angle as Mongo had and. Mongo, I think Postgres had a really good answer too with Jason B. Supportive a little while later. and that I think was, was an excellent move. I'm really looking forward to seeing what we can do in the [00:19:00] Postgres ecosystem kind of this time and see if, see if that, um, I think, cuz I think Postgres is built on better fundamentals than one has a lot of features that, and that I love. So I'm really interested to see how we address that.

Nikolay: Yeah. And when you talk about some start new startups, startups, uh, which choose with TES and my sequel, is it like very, very new startups or is some.

Michael: new this this year or so. Yes.

Nikolay: interesting. Interesting. Well, okay. I, I don't see such data, I see constant discussions about like, oh, it would be good to have it as, and it has, uh, I think later, last year, like a year ago, they announced that they don't have plans to work on Postgres, support themselves open to community and contribution. But, I don't, in in my head, I don't see, maybe like I have some filters. I don't see, uh, companies who, who these days will choose my SQL without, strong pressure from, I don't know who, like, [00:20:00] it's, it's strange, but I. I can't imagine, like if, for example, some CTO wants to be protected in terms of risks, to grow growth risks and how to handle growth. And, for this CTO, it doesn't matter that, uh, you can handle dozens of thousands of transactions per second using single simple cluster or on modern software actually in already hundreds of, uh, transactions. And I, when I say transactions, I don't, mean transactions, of course, I mean, Social media, like, traffic, like 90% are seller, right? Or 80%, but others are rights and you can handle perfectly. You can scale up to. like you, you can handle hun hundred thousand TPSs or few hundred thousand TPSs, especially if you take for example, modern, uh, epic, uh, uh, epic, um, processors from amd, epic Milan, epic Rome as well. So aroma's, previous generation, Milan, a [00:21:00] lot of CPU power. Very, like a lot of memories. So, and you, you, you run. 10 notes and cluster a lot of replicas. It can be very, very, very powerful. So you can be, you can postpone. case when you need to be to be shared. And also microservices is a, like, people sometimes choose a microservices and postpone the need to, to chart their databases, right? Because databases become, become smaller. So yeah, I can imagine the reasons behind the choice, uh, of, uh, to choose, uh, test. But is it test or plan scale, by the way, speaking of managed services,

Michael: They are, the startups I'm seeing are choosing planet scale. They, I don't even think they would know that. They wouldn't necessarily know what SS was. I think.

Nikolay: it's different because as I see with Plan scale now has two value propositions combined. Uh, one is, uh, uh, Webscale [00:22:00] plain scale charting. And another is we will handle your changes without problem. They call database branching, but it is not. but they provide you like ability to change schema and then have some similar to. Pull, pull, request, merge, request concept, deploy request. So your colleagues review change and then you, they run change, , online without downtime. and you don't think, like you don't need to spend efforts. This may be also a big motivator because pogs in pogo, it's a big headache to, to avoid, locking issues and so on.

Michael: This is what I meant by their developer friendly features, and I think that's a similar value proposition that that Mongo were offering back in the day as well.

Nikolay: Mm-hmm. . Mm-hmm. . Yeah. Interesting. I think many POGO companies, positive select companies also already working on this because, uh, of course. Yeah. So in this case, let's choose, let's proceed to next item number four, and let's, this is database branching is coming, [00:23:00] right? So database branching is coming and originally, I think last. It was plenty scale who started to use this term and in my opinion, in the wrong direction because they talked about, branching schema only. So they're branching. Data based. Branching is just branching of schema. recently they added. So-called data branching, which is something strange to me, but it's another story. But also, several other companies in POGO system because we are mostly interested in POGO system. They started to talk about database branching, and for me, of course, This is very close to home, right? Because I, um, my company develops database lab engine, which provides thin cloning. And actually yesterday we released First Alpha, which supports database branching. And branching is not cloning. There are differences. Another story, but. I'm glad that many companies already think in this, uh, in this direction. They just want to work with, [00:24:00] like they original, uh, need is to be able to work with databases and non-production, with bigger size databases. same as with gi. So you have independent, database. You, you can have multiple independent databases and you can experiment, develop, test, and so on. And cloning, which we had for a couple of years already supported it. But, branching, uh, allows you also to. have some progress and committed, like to have snapshot and then share with your colleagues. They can branch from there. So it's like nested cloning already. We've, uh, nested cloning with, uh, comets, it's branching. This is, this is what, what happens if, if you look at gi, it's very close to it. and I would like to mention Neon, which uh, released branching a few weeks ago, and, uh, it's also already publicly available. It works very well, I think they don't have committees yet, but I'm sure they [00:25:00] are already thinking or working on it. Bit natural. But, overall this year, I think something like big, uh, movement in this direction started. So I predict in, in future few years we will see a lot of progress in this area. So, uh, a lot of roadblocks in development and testing. When we don't have big database to play with, when we need to develop or test, it'll be, Available to most teams, I think in future, thanks to several companies who work, who work on this area, including mine. Of course. So

Michael: I've seen, super base also mention branching, in, I think maybe in one of their

Nikolay: the roadmap, in yeah, in their roadmap it exists. Yes. Yeah.

Michael: And, uh, other, other Postgres, related things in this, in this area, I would've expected if Heroku had kept investing in their, or if they'd kept innovating on the lines that they were [00:26:00] going, I think they would've done this.

Nikolay: they have, , very good concept, so-called, um, preview, uh, preview apps, right? It's basically environments, which. Um, deployed by, by request, explicit request or inside C I CD pipeline. So for imagine the, for, for each branch when it's, when you have GI push, C I C D pipeline is running and, uh, Heroku can deploy. Preview app on, on specific, uh, url, and it has, uh, adjusted code from that branch. But there is always a problem, what to do about databases and naive approach is let's have one database for all, but there will be conflicts of course. And, uh, if you want to de to delete something or to change schema, especially. like a lot of conflicts. Uh, and, branching database. Branching and think loaning. It's exactly what can feel this idea to like [00:27:00] complete state. So, so it can, you can have, uh, preview apps or like environments, uh, as a service, not only in jobs of coal, but also in terms of data. And that's exciting. Like it's super cool.

Michael: Yeah. And then it seems like, and I, uh, the natural successor to Heroku Postgres is gonna be Crunchy Bridge. They've, uh, got a lot of the same team involved. Uh, so I'd be interested to see if they're planning to do anything on this

Nikolay: Yeah, it's interesting. I I, I, I didn't see anything about it in this direction yet, but maybe there's something I would like to know to learn about it. Yeah. But anyway, I, I see. Okay. We, uh, like obviously several companies look in this direction. I, I hope finally this problem will be solved in next five years because it's something that should be solved. and, uh, it'll unlock problems with, like, for example, some, some product manager want to, wants to try it with data, right? Or you [00:28:00] develop multiple things simultaneously, or, or you have multiple developers or you even if you want to optimize, uh, sql. Uh, to check performance. It's also a way to go because without data, performance is different. So you cannot, check your, your performance if, if you don't have enough data. And this unlocks many, many things like that. And eventually, it'll, it shifts a lot of staff to left in terms of the ops, uh, infinite loop. So it's definitely shift left testing, shift left, uh, activities. I'm excited about this.

Michael: Yeah. if anybody's new to our podcast, we do have an episode specifically on database branching, so we can, we can link to that as well.

Nikolay: We have two episodes, I think. Right? So from time to time with topic, this is my favorite topic. So I think we will continue doing this, I hope, right? Because there are interesting things about performance testing, for example. Okay. And my last item from my top list, top [00:29:00] five list is actually paulus, we, we mentioned that in the. POGS is everywhere. all cloud vendors provide pogs and even Oracle started to provide managed POCUS service. So like I, I knew about, um, SAP for example. It was like few years ago. So every, everyone, literally, everyone. And for example, also interesting news like Google, released. database cloud database called Allo db, which has something from po. like of course, it's like, can be considered as from pogs family of database systems. And it's also super interesting. It has very interesting concepts, to handle h step, uh, hybrid, uh, transactional, uh, processing. So it's like an analytical and operational processing, uh, combining. They, they have a combination of, a roll store, cone store memory. So like, it's interesting, I, I haven't tried it yet. I still my to-do, but, sounds super [00:30:00] interesting. And overall, as I've said already, we have dozens of options to choose from. If we, okay, you need s, but which one? So many. Right? So to choosing right option is, uh, is not simple. It's not a trivial task. It's not like before you download binaries or you download source code, compile. Make my install and you're done, right? No, these days we have so many options. It, it's not only like, managed and versus self-managed. If you say managed so many options and self-managed as well. There are many Kubernetes separators already, like maybe five quiet active projects, which, uh, all of, I'm also interesting. More than five, actually. More than five. There are other reviews of them. , if you want to self-manage question will be old school self-manage or Kubernetes, right? Because there are many operators and they provide, they bring things like backups, [00:31:00] replication, monitoring, uh, out of the box. They are. Eventually, we will compete with managed services and managed services. Should be afraid of, uh, these operators of. so, uh, this is, this is it with my list. Do you have something to add?

Michael: No, that was, I think that was great. I, a couple of smaller things that probably don't, weren't a big discussion, but I thought it was worth an extra shout out to the team behind the Postgres 15 features. All of the people that contributed to that. I know it was more of a, kind of a lots and lots of smaller features this year. Uh, but I re really liked some of.

Nikolay: who mentions this? We know, we all know every, every year we have new podcast version. It's, it's not a news already, right? ? I'm joking. I'm joking. There's there and other things. Right?

Michael: yeah. But I don't wanna take it for granted either, right? Like, it, it is every year that we do have a major version, but that's not necessarily guaranteed to happen forever. , it is really cool that we do, Also, I think [00:32:00] there's probably not so much for this past year, but I think next year I think we'll see some really interesting serverless, or, I don't really like that word, but that seems to be the standard word. Um, use cases like if, People can use Neon to, to send, uh, Postgres queries to in enormous serverless nature. I don't, don't understand how it works, but that could be really cool, I think. And then the other interesting project I'm keeping an eye on is oil db, which looks like it, the,

Nikolay: Oh, oh. B also has branching in the roadmap. and they have, uh, there's, uh, in, in GitHub there's a file, uh, like markdown file describing the concepts, and they already have copyright checkpoints. So if, if they implement database branching, it'll be po native database branching. So you don't need to think about separation of storage from compute. You don't need the first, like we do. You have it inside positive. It's, it's, I'm, I'm super excited about this.[00:33:00]

Michael: Yeah, very exciting. but yeah, I think you've done a great job of wrapping it up. It's been, I think it's been a really good year for post Chris and looking forward to another one next year.

Nikolay: I agree. I agree. Thank you. Well, everyone have a good new year. Much better than last one, this one. And, see you soon. Thank you so much.

Michael: Absolutely. Happy New Year. Bye now.