Breaking the Pillars: Rethinking Observability with Charity Majors

Most observability platforms are built around siloed signals, metrics, logs, and traces; not the rich context engineers actually need. The architecture of observability tooling hasn’t kept pace with the architecture of today’s distributed systems. 

In this episode, David sits down with Charity Majors, co-founder and CTO of Honeycomb.io and co-author of Observability Engineering, to rethink how modern teams understand production systems. Charity argues that the “three pillars” model is outdated, making the case for wide, structured events that preserve context and embrace high-cardinality data so engineers can ask better questions in real time.

Drawing on lessons from operating Parse  and its challenging acquisition by Facebook, Charity explains how those experiences shaped Honeycomb’s fast, custom-built observability engine. The two also  explore how AI agents could soon close the loop between code changes and production validation, and Charity previews the upcoming second edition of Observability Engineering, including a new focus on observability governance for CTOs. 

Learn more about Charity:

• charitydotwtf.substack.com

• honeycomb.io/blog  

• x.com/mipsytipsy

Transcript

00:00:03.320 — 00:00:06.200 · David

All right. So welcome to the podcast Charity. How are you doing today?

00:00:06.320 — 00:00:08.640 · Charity Majors

Yeah, I'm doing good. Good to be here.

00:00:08.680 — 00:00:17.320 · David

Yeah. I was just complimenting you on how cool your setup is, like, right before we joined, like, look at all those colors, everybody. It's it's fantastic. You know.

00:00:17.520 — 00:00:21.640 · Charity Majors

You see the little, uh, 3D printed honeycomb lights.

00:00:21.680 — 00:00:23.400 · David

Yeah, I'd see that. Yeah. That's cool.

00:00:24.080 — 00:00:28.000 · Charity Majors

I need to, I need to get I get an extension cord so that it's all lit up.

00:00:28.040 — 00:00:28.680 · David

It's all lit up.

00:00:28.720 — 00:00:29.800 · Charity Majors

Yeah, I love it.

00:00:29.880 — 00:00:31.480 · David

Do you do custom make that.

00:00:32.040 — 00:00:36.960 · Charity Majors

I did not custom make that. Okay. But, uh, one of my coworkers did.

00:00:37.000 — 00:00:49.280 · David

Got it. Yeah, I know a lot of people today are getting into, like, 3D printing, and I see, like, all these YouTube videos, and I think my Instagram jam is now like a bunch of people making, like, so many cool things. I think it's amazing the era we live in today.

00:00:49.360 — 00:00:50.240 · Charity Majors

I know.

00:00:50.800 — 00:00:57.240 · David

Yeah. Ever wondered if you could go back in time and like, drop a 3D printing machine? How is it change human society?

00:00:57.960 — 00:01:01.620 · Charity Majors

I have not had that particular thought, but now I do.

00:01:02.060 — 00:01:17.260 · David

Yeah, well, welcome again to the podcast. I hope you're having a great day. For, you know, for everyone listening in, uh, you know, Chidi, uh, why don't you kind of give everyone a little bit of an intro of what you do today? Um, and, you know, your role at honeycomb and things like that?

00:01:17.580 — 00:01:49.500 · Charity Majors

Sure. I am the co-founder and CTO of honeycomb. Io, uh, the original observability company. Um, I'm also the coauthor of Database reliability engineering. Observability engineering. And today, the very last chapters of the second edition went out to tech reviewers. So next edition will be out in.

I think Dead Tree Edition will be out in June, but chapter will be coming much sooner. So. Nice.

00:01:49.660 — 00:02:38.770 · David

Very cool. Yeah. It's awesome. It's an honor to have you on the podcast, actually, and I'll give you the background. Okay. And like, it's always great to listen, uh, to somebody and say, hey, this is what I do. And it's like, you have to, like, condense your entire life into, like a 15 second intro, which is, like, completely unfair.

But let me tell you my story of charity majors, okay? I used to work at a company called Data Stacks before I joined Cockroach Labs. All right. And you are pretty familiar with it? Yeah, I used to work at a company called parts, which in 2012, 2013, if I don't remember, if I recall correctly, it was suddenly became like the poster child of Cassandra.

Yeah. I think. Is Christine still your co-founder for honeycomb? Yeah. Okay, cool. So I think Christine wrote, uh, or was Behind Bars Analytics, which used Cassandra, and I think we were at Cassandra two or something or one, and it.

00:02:38.770 — 00:02:40.530 · Charity Majors

Became the whole thing.

00:02:42.250 — 00:03:25.950 · David

Yeah. Yeah. So I was early on, before I joined data stacks, was like following Cassandra as a open source database. And at that time, uh, and, and then I saw that parse was like one of the babies. And then obviously, uh, Cassandra came out of Facebook. Powers got acquired by Facebook. And I think this whole story.

Yeah. So and I saw I would see charity majors here and there and then when I joined data stacks, I mean, I think there was a point where we had a cross section where I read a lot of stuff that you wrote, and I think by the time you had started honeycomb. So that's my background. And it's so it's so surreal that somebody who's content and blogs and, uh, you know, information you read about, you get to talk to them.

It's an honor. Actually.

00:03:26.030 — 00:04:11.850 · Charity Majors

This is amazing I know. Yeah. Thank you for telling that story. Yeah. Know that that pass analytics product in a, in a very real way is responsible for honeycomb and thus observability. Because Christine, like we had this whole origin story. Right? Christine. The dev charity, the ops, you know, Christine single handedly built pass analytics on top of Cassandra using all the best practices.

but when you're using metrics, you have to decide upfront what questions are you going to what people ask. And our past users kept writing in and being like, but I want to ask this, but why can't I ask this? And Cristina just be like, oh, that's frustrating, you know? Yeah, yeah. And I was I think I was anyway.

00:04:12.130 — 00:04:30.690 · David

Yeah. I mean it's, it's, it was an era. I mean I think like yeah, it was an era for sure. I mean, I think people, people who I mean, whoever listens to it who remembers this would know how, how crazy it was. Like, I remember I was I think I write a blog at the time where I think parse was like blowing up so much that like, you had probably a million apps running and.

00:04:30.690 — 00:04:37.650 · Charity Majors

That million mobile apps hosted on MongoDB. Yes. Literally yesterday on X, I,

00:04:39.210 — 00:04:51.010 · Charity Majors

uh, the founder of I won't name it, but was was tweeting about how the the old way of understanding your software was you read the code. The new way is you read the. And I'm just like

00:04:52.070 — 00:05:24.270 · Charity Majors

In what world did you go? I read the code. I understand what it does. Well, it purrs, you know, at Paris and at Heroku, we were hosting developers code, so we had to understand it from first principles and instrumentation. We couldn't just go ask the developer what they were trying to do when they shipped it, you know?

And that was the whole mindset. And I forget that's not everyone's mindset. The primary debugging tactic that most teams have to this day is who shipped that day.

00:05:24.390 — 00:05:25.270 · David

Right, exactly.

00:05:25.310 — 00:05:35.230 · Charity Majors

Go find them. Ask them what it's supposed to do. Get them to fix it. That's the fastest short. You know, and like, I'm like, I haven't been in that mindset for 13 years, you know?

00:05:35.550 — 00:05:53.620 · David

Yeah. No, I mean, I think it's what if like, what you're saying is very interesting because at that time, I don't know where it became true that I think it became a standard in many ways. Everybody was trying to say, this is a straight standard, Kubernetes standard, the container standard, the observability standard included logging metrics.

00:05:53.900 — 00:06:04.940 · Charity Majors

Tracing failures, metrics, logs, and traces. Well, except for exceptions and errors, and every signal type go to its own silo. Talk to this guy.

00:06:05.180 — 00:06:05.940 · David

Yeah, exactly.

00:06:05.980 — 00:06:55.760 · Charity Majors

Yeah, but like that, I think people don't realize that's not a best practice. That's an artifact of operational stack infrastructure. You're mostly operating code that you can't change. So you just have to take whatever it emits. If it sends you metrics, you just gotta put it someplace. If it sends you logs, you just got to put it someplace, and it's shitty.

But it's what you got, right? That is not for your crown jewels. For your software. The software that makes you money. That's not what you want to do. You have power over this code. You should instrument it in a way that does not spray your signals apart. That keeps it together because the context is what makes that data powerful, right?

00:06:55.800 — 00:07:44.640 · David

Yeah. And I think observability is like has become so much critical. And I do feel with the agent scale at which things are coming, where we are going, uh, it's going to become way more critical. So I would love to dive into your perspective on all of that in this episode. And for everyone listening. And obviously, you see, like the small connection that I've had with Charity from from far.

I was, by the way, this was era. I was not even in the US. I live in the US now. I was in India and I was read these things like find it on the internet. And communities used to be like so weird at the time. We have so much more organized now, so it's great. But before we get to parse, uh, tell us a little bit about young Charity.

And you know what? What really excited you to pursue this path? And, you know, how did this happen?

00:07:45.000 — 00:08:04.739 · Charity Majors

You know, I grew up in the backwoods of Idaho. I was homeschooled. Um, religious compound, basically. Um, and I got a piano scholarship to college when I was 15 and knocked out. I had never used computers and

00:08:05.740 — 00:08:31.340 · Charity Majors

my first year of college, and I was just like, dude, I grew up poor. I am not. Nope. Nope. Uh, and, you know, I got a job as assistant then, and I've been on call since I was 17, you know? Uh, traded one keyboard for another. But that wasn't it. That was an era when you could just get into computers as a tinkerer.

00:08:32.099 — 00:08:52.250 · David

Uh, so. So you did that. You worked as a tinkerer. You worked at pars, and then eventually, uh, join Facebook. Uh, got hit by a completely different scale. Uh, tell us a little bit about how that parse into Facebook experience was, and how you got experience with scale and observability in start of everything there.

00:08:52.290 — 00:09:09.090 · Charity Majors

You know, so I had been at Linden Lab during the there was a time place where Facebook seemed like the redheaded stepchild and like Second Life was seen as like the next big thing, right? So the scale wasn't super unfamiliar to me, I.

00:09:11.410 — 00:09:55.789 · Charity Majors

Acquisitions are hard. They're always hard. Like culture clash. They're kind of nonconsensual. Like, I didn't go to work for Facebook for a reason. And then my company was like a two hour bus ride each way. There was there was a lot of culture clash and a lot of things that I, I was always one of those engineers who didn't care about the business stuff, didn't want to hear about it, didn't hear it right.

I was a backend infrastructure. And that too, I think is an error that is passing, and people who feel that way should be mindful that it's not a growth, it's not a growth area. I wish I had paid a lot more attention to business outcomes and

00:09:57.510 — 00:10:32.830 · Charity Majors

product development and design and, you know, that sort of thing. But we got acquired by by Facebook. And, uh, a couple years later they shut it down. Um, I left before they said, but ultimately, I will say tough experience, but I'm very grateful to have had it. I well, I've often said I don't think I learned anything new about technology while I was at Facebook.

I learned a lot about management, about like, you don't get to be a company that has tens of thousands of people without learning how to operate.

00:10:32.830 — 00:10:33.950 · David

Well, right.

00:10:33.990 — 00:10:34.430 · Charity Majors

That's what I.

00:10:34.430 — 00:10:51.850 · David

Learned. Yeah, yeah, it completely makes sense. And I think, you know, when I was researching also, I kind of saw that that was your pivot to go from, hey, this is I need to go learn or figure out what I've done at this company and apply it at another company. And I think those were your beginnings, honey.

00:10:52.010 — 00:11:39.750 · Charity Majors

You're applying more intention than the process. I love Facebook. Looking around for the next thing and I have the like VCs were like, would you like slow? I'm like, well, I'm not gonna say, yeah, sure. Like as a woman, a dropout, uh, queer person. Like this opportunity. Of course I'll take it. I'll try it.

We're definitely going to fail. But, like, you know, it's a fun way to spend a couple of years ops. People don't usually start companies, and VCs don't usually fund them. And for very good and obvious reasons, because we're not optimists, we expect everything to fail. That's not really the outlook that you want in a CEO or a founder.

You want the people who are like, this could be $1 billion business someday. And yeah, I would never meet, so I do agree.

00:11:39.790 — 00:11:54.670 · David

Yeah. But I, I think like when I was like. Like looking at what honeycomb started to do and as, and I think some of the new content that I'm seeing, it's like so exciting. Like it's just a different way with which you are looking at observability. I don't even think people get it.

00:11:54.710 — 00:11:55.910 · Speaker 3

So no they don't.

00:11:55.950 — 00:12:12.150 · David

Correct. So so walk us to what you saw as to like what was the problem and then why you felt like, hey man, we need to do something completely different and project a different way to help the way the world is going to be. And I think you're like Prime now with what's going on with all the AI.

00:12:12.310 — 00:12:14.950 · Speaker 3

We were built. Very exciting. We were built for this era.

00:12:14.990 — 00:13:06.180 · Charity Majors

We didn't even know it. But we were built for this era because what people are struggling with the most right now is not AI or non determinism or the novel aspects. They're struggling with the software aspects, but it's ten times as much, a hundred times as much, ten times faster than before, 100 times faster than before.

Right. Like their systems have evolved to accept a rate of change like this, right? And it's held together with duct tape and string and stupid human tricks like intuition. And I've been there for ten years, so I built it so I can guess where the problem is. And as that rate of change ticks up, I orders all the duct tape is flying off the bus and honeycomb.

You're absolutely right. We are. We are not really descended from the mainstream.

00:13:06.180 — 00:13:06.460 · Speaker 3

Tools.

00:13:06.500 — 00:13:08.379 · David

The traditional observability

00:13:09.420 — 00:13:10.300 · David

cloth or something.

00:13:10.340 — 00:13:11.660 · Speaker 3

Yeah, yeah, we.

00:13:11.700 — 00:15:36.929 · Charity Majors

We were the ones who introduced the concept of observability for software, and then it got co-opted by all the other companies. But we had the DNA of honeycomb is much more, um, like a combination of product analytics, business analytics and, you know, just systems. It's just arbitrarily wide structured data blobs that can be hundreds of dimensions wide.

They can also be traces because traces are just structured logs with IDs, you know, and that packs all the contexts so that you don't have to decide what questions you're going to need to ask in the future. At the time that you're storing it. You can you can make all those decisions at some, which is exactly what you need to understand your AI.

You need context, speed and flexibility, the ability to slice and dice and zoom in and do mountain. That's what you need. That's what you need. That's what we built, you know? So it really is. I give a keynote to our company last month, and like I said, I'm not an optimist. I'm not an early adopter. I've seen hype trends come and go many times in tech.

The last few were a bust. So like, it really wasn't until last fall that I realized, guys, This is our chance. This is our opportunity. This is the moment we've been waiting for. For. For all of any comes existence. Our our essays, our blogs, our thought leadership have had a lot more reach and influence than our product has.

And it's always puzzled me. Why is it people out there who love what we say, love what we write? Why don't they then immediately go sign up for the product? Because it's the same thing, right? The product encodes our philosophy. Our philosophy comes from our product. They're the same thing, right? Yet there are people who have been listening, reading our stuff for ten years.

We're like, oh, it never occurred to me to try honeycomb, right? Well, I think it's because the delta between where they are and where we're talking about is wide and it's come across like, eat your vegetables if you want to do it the right way, do this instrument, your code this way. Look at your graphs after you do all these tests in production.

You know, all these things. And

00:15:37.970 — 00:15:39.770 · Charity Majors

now with AI,

00:15:41.210 — 00:15:59.250 · Charity Majors

the it's competitive pressure. It's not us saying eat your vegetables. It's oh, shit. We're not going to actually reap any value from our AI investments unless we can build the guardrails so that we encode the knowledge of the system into the system instead of into, you know, intuition and all these other things.

00:15:59.290 — 00:16:33.710 · David

So you've said some really cool things. And I think, I mean, as I was saying, my point was really like when I was researching honeycomb, I think the philosophy and the experience that you bring from the I, even the observability engineer book that you wrote right in the second version that's coming, it's it's so rich in looking at something that everybody thinks is a is the way you should look at, but you kind of have opened it up in a much more wider way.

So the perspective that you were bringing with the AI agents and things like that, if you do, you see like the scale at which AI agents are growing is going to just break existing observability.

00:16:33.910 — 00:16:36.310 · Charity Majors

Oh, yeah. Yeah, the three pillars model.

00:16:36.750 — 00:16:37.190 · Speaker 3

Uh.

00:16:37.590 — 00:17:33.569 · Charity Majors

Yeah. It doesn't it doesn't work. You know this. And this is not just me as a vendor. Like being like, oh no, we hear this from customers and prospects, like, constantly. Like when we're trying, we're trying. It just doesn't work because that connective tissue, the context is the most important part of the data.

And when you're splitting your signals up into different silos, you're discarding it, right? Data becomes more valuable with more context, and it becomes not linearly, more valuable becomes exponentially more valuable. And so when you're like, we're going to put the metrics over there and put the logs over there, we're going to put the traces over there, you know, and then like I think existing vendors are trying to kind of monkey patch all this by then building bridges between those pillars, which and this is why if you look at sort of like cost, which is most people's entry into their just like

00:17:35.130 — 00:17:41.490 · Charity Majors

cost has been rising 48% per year year over year for the past 15 years straight.

00:17:42.570 — 00:18:17.050 · Charity Majors

Like Gartner printed something, uh, talked about, a representative customer of theirs in 2009 was spending $50,000 a year on monitoring. And in 2024, they spent $24 million. And it's continuing to go up at that rate. And that's because you're storing the same data again and again and again and again in different formats.

And then trying to build bridges between these data types and, and and it doesn't work. It just doesn't work like it. It just doesn't work. You want your data to be you can't put Humpty back together.

00:18:17.250 — 00:18:18.490 · Speaker 3

Yeah, absolutely. Yeah.

00:18:18.530 — 00:18:32.070 · David

And so with that, I mean, this is a great segue to the question I wanted to ask is that how is honeycomb doing it differently compared to these other vendors. Like would be great to know a little bit of what your vision is and how that has been applied to the product.

00:18:32.110 — 00:18:36.950 · Charity Majors

It's not rocket science. Like we're not that smart. Um,

00:18:38.630 — 00:18:46.230 · Charity Majors

back in 20 years ago, uh, columnar stores, 20 years ago, a couple decades ago, um,

00:18:47.950 — 00:19:46.500 · Charity Majors

they became productized for business teams, right? But they were super expensive. So you had to be very selective about what data you would put into that. But that's what powers your sales teams, your marketing teams. You know you can't. Yeah. They couldn't do it if their signals were all spread out. It's really only in software engineering that we're just like, ah, I can hack this together.

You know, software engineers are using hand-me-down tools from ops teams who have to use those tools because they can't modify the code. Right. But it's just it's a columnar store. um. It is. I will brag a little bit. Uh, we do have by far. We have the best storage engine in the world for observability workloads.

It's the only one that's custom built for observability, which means that we can make. We couldn't power, you know, we could power Facebook scale workloads, you know, because it's horizontally partitioned and distributable. And we can

00:19:48.060 — 00:20:57.680 · Charity Majors

like that if you want to run a query, our 95th percentile, I think query response time is under a second. So it's interactive. It's fast. You know, it's just because one of the problems with the old model is you got to index and schema and make decisions about what you want to be able to ask fast questions about in the future.

Problem is, if you can't predict what you know, the whole model. But with a columnar store, everything is an index, right? And you can pack as much context you are up to, like the limit of the number of Linux file handles that can be opened in the system, right? So it's like thousands, right? So there's no penalty and there's no penalty for cardinality.

Right. A lot of companies are paying out the nose because high cardinality data is the most identifying data in your system. Unique IDs, names, you know, descriptions, super high cardinality. And most companies are just discarding it because they can't afford to keep it. Um, so just again and again and again, we're built for this.

You know.

00:20:57.720 — 00:21:36.940 · David

No, it's it's very cool. And I mean, we at our company, I was recently talking to our CEO, and, you know, we were talking about the words that we anticipate. And I think one of the things that we have started to recognize is that there is a limited amount of power available and there's a limited amount of data centers available, but we are actually trying to expand on that.

What that signals is that there will be more and more people trying to fetch data, read data, write data, do all these different things, and you need to have a way to observe all of this. And so the joke is, where are we going to put these data centers? Obviously some people say we claim we'll have data centers in the space.

00:21:37.140 — 00:21:38.860 · Speaker 3

And maybe, maybe on them.

00:21:39.260 — 00:21:43.220 · David

Maybe. Yeah. I mean, I mean, that's what that's the whole. Elon Elon has been talking about.

00:21:43.260 — 00:21:47.660 · Charity Majors

I just I cut out my Elon Musk feed a couple years ago.

00:21:47.700 — 00:21:47.940 · Speaker 3

Yeah.

00:21:47.940 — 00:22:11.500 · David

But regardless of that I think it's very clear. Like I recently got to know one of the big data center investments is happening in a port back in India. And like just data centers are exploding everywhere. But I think if you go back to the root of it is this AI scale that you're talking about, like when somebody I was talking to recently told me he's running, that individual is running 16 agents on 3 or 2 Mac machines.

00:22:11.540 — 00:22:13.060 · Speaker 3

And this is where I don't I.

00:22:13.060 — 00:22:17.620 · Charity Majors

Think the interesting questions are less about scale and more about the rate of change.

00:22:17.620 — 00:22:18.450 · Speaker 3

Right?

00:22:18.530 — 00:22:26.730 · Charity Majors

I mean, they're both interesting that they're there's so many. But like, the things that really make me lean in and go, ooh,

00:22:28.050 — 00:22:30.170 · Charity Majors

I think it's the rate of change.

00:22:30.370 — 00:22:31.090 · Speaker 4

Yeah, exactly.

00:22:31.090 — 00:22:54.570 · David

And, and so I was asking him like, so if you're running all these, if you're making all these different changes and if you're running all these different agents, uh, how are you handling everything else that needs to support it? Like, so I was asking him, what kind of applications is your agents hitting?

I was telling me, like, four of his agents do four different tasks, and he's basically broken their context but hit the same application.

00:22:54.650 — 00:22:55.090 · Speaker 3

Yeah.

00:22:55.130 — 00:23:37.430 · David

And and I was like, how? How what? I was like, curious, like, how do you identify them? How does it go into your database. Do you track it anywhere? How are you observing it? Nothing. He's like, no, I'm just something breaks. I go back and try to fix it. I try to figure it out. So even at a very, very low level, I'm seeing like every small, every day individuals are getting into this and citizen developers are growing.

So what would the impact of that be on, you know, Facebook like companies who might open up a gateway. Okay. Interact with agents. So like of course the scale is there. But the obviously the human behavior is changing with respect to age. And so it's very interesting. So I was curious as to what you feel about that here.

00:23:37.750 — 00:23:49.390 · Charity Majors

Boy that's a big what do I feel about the intersection of AI scale, rate of change, developers and big companies surfacing agents for us to talk to? David.

00:23:49.910 — 00:23:54.310 · Speaker 3

Right. It's a question that doubts. Yeah. Yeah. No, no. Makes sense.

00:23:55.390 — 00:24:13.590 · David

Right. Right. Yeah. No. My point of view is like I feel like this will this will have massive implications to performances. Right. And then performance is engineering and you know, system design will become so much more critical. And you cannot do system design without looking at how to tackle the observability.

Right.

00:24:13.630 — 00:24:23.810 · Charity Majors

So this is something that I. Something that occurred to me a couple of weeks ago is right now we are in the process of starting. We are building the first

00:24:24.930 — 00:24:27.490 · Charity Majors

AI native sociotechnical systems.

00:24:27.530 — 00:24:28.330 · Speaker 3

Nice.

00:24:28.650 — 00:26:01.670 · Charity Majors

You know, none of us are AI native like three years ago, right? It just came. And now we're starting to build AI, AI natives by looking at, okay, what do we what lessons can. There are a lot of analogies to the cloud. You know, uh, I in fact, I wrote a piece on my Substack called something like 2025 was the AI. What 2010 was to the cloud?

You know, the year that it stopped being a toy and people's the center of gravity shifted, right? It went from, oh, this is a the cloud is a place where we can put our extra workloads to, oh, uh, everything else is legacy. And we need to start building, you know, for the cloud. And so what are the patterns? What are the new patterns.

Right. What are the new assumptions? What are the architectural details that we all thought were just like baseline that were specific to on prem environments. And and I think this is really interesting. And to your point exactly. Um, I think one of the lessons that we have learned or should have learned, I don't know, one of those two, uh, is that data is made powerful by context, and you need to keep it together, preserve the relationships in your data storage that that do exist in the real world, right?

Because that's that's what AI needs. Preserve the relationship. Some kind of ontological mapping to reality, semantic conventions, you know,

00:26:03.070 — 00:26:03.910 · Charity Majors

and

00:26:04.990 — 00:26:22.220 · Charity Majors

the data thing and the observability thing if you can. And and some people are like, ah, but it's a black No, you can read the trace. You can? Yes, it's somewhat non-deterministic, but you still you're not exempted from trying to understand it if you're trying to make it better.

00:26:22.540 — 00:26:22.940 · Speaker 4

Yeah.

00:26:22.940 — 00:26:27.180 · David

And I think one of the things is like people are bad at observability.

00:26:27.300 — 00:26:27.900 · Speaker 3

Oh, yeah.

00:26:27.940 — 00:26:49.100 · David

Like, I mean, that's the point. Like, I mean, I mean, if you go back to some of the stuff you have written in the past and this I was reading it and I think one of the things that came out was that it's not natural for like where humans can observe stuff, but we are not good at observing stuff that's breaking all the time and like being at anticipating patterns.

So you want to develop that a little bit?

00:26:49.140 — 00:26:49.940 · Speaker 3

Yeah, I.

00:26:49.940 — 00:27:16.400 · Charity Majors

Think historically this has been kind of it's a running joke in SRE that everyone in SRE has ADHD, like almost literally. And everyone in Sri is someone who, through personality or training or both, Immediately looks at what will fail. It's just like you showed me a cool idea and I'm like, ah, this is going to fail.

And it makes this really fun at parties. Like,

00:27:18.360 — 00:27:25.040 · Charity Majors

there are a lot of people who find talking to us about their ideas deeply frustrating. And I have really had to

00:27:26.840 — 00:28:24.900 · Charity Majors

personality, retrain, transplant myself a bit because it's just not always helpful. Um, but yeah, we don't as human beings, most people don't immediately focus on, okay, how is this going to break, what can I anticipate and so forth. And there's been quite a feedback loop in the past because, um, people didn't actually have the tools to do that.

Well. So even if you did anticipate and you did instrument and all these things, if all you had was dashboards and aggregates, there's kind of no point, you know, could you couldn't you couldn't really you could tell if something really big and bad happened, but you couldn't understand precisely the impact of your change.

And and one of the reasons that, you know, a quote I've been trotting out a lot lately is that AI is like alcohol. It is both the cause of and solution to most of life's problems.

00:28:24.940 — 00:28:26.180 · Speaker 3

That's right. Yeah.

00:28:26.220 — 00:28:43.780 · Charity Majors

I would say that one of the Eat Your vegetables messages that I've been saying for like 15 years is instrument your code as you write it, commit your code, then go look at your instrumentation to close the loop. Understand it. Right. But

00:28:45.260 — 00:29:36.490 · Charity Majors

if you instrument your code and you go and you look at it and all you have are like, okay, 99.8, you know, with AI we now have the ability to auto instrument using hotel and open source patterns. You know, it's better than we can do ourselves and build that loop into our code so that it can come in and tell us, oh, that endpoint that you shipped yesterday.

Errors are super elevated for that. You know, it can. It can become a conversation with your code. And if you use something like honeycomb, is it not true of three pillars? Just to be clear. But you can do this with precision. And so like what I am excited about with agents in particular, is the ability to bake this loop into the product and into your workflow so that.

00:29:39.730 — 00:30:28.990 · Charity Majors

I guess I'm excited. Mouth not as fast as brain, but like you ship a def, right? You have that intent in your head. This is the intent you have. Okay, so you generate code, you generate tests, write, test the intent. Cool. You deploy to production where you have things like progressive deployments and feature flags.

You can test safely in production. So and then your agent goes and looks precisely at the change that you made. What is your intent? What do you expect to happen? What is the comparison to baseline on all the dimensions? And it can let you know. Looks good. Okay, let's roll it out to a 10% of hosts. Right. And every step that can.

Is doing what I expected it to do is anything else. You know. And so you can promote it and then imagine, you know, that whole thing about.

00:30:29.030 — 00:30:29.390 · Speaker 3

Never.

00:30:29.390 — 00:30:33.310 · Charity Majors

Deploy on a Friday, you're abusing your team's if you're blah, blah, blah, blah, blah.

00:30:33.510 — 00:30:33.830 · Speaker 3

Yeah.

00:30:33.830 — 00:31:38.130 · Charity Majors

That's a that is a trope. If you if there's no if you can't understand if you can't see the impact of your change. Right. But if you can see it precisely if you have confidence that you can see the delta, you know, you you in order to move swiftly, you have to have confidence. Confidence is what allows you to move.

So anyway, agents, you know, imagine you just shift you. You just shift a little feature, right? And your little agent's going to check up on it and let you know if something bad happens. Precisely. Because even if it's, you know, say you're you've got a, you're running a system that's shipping a million requests per second.

And this change that you shipped is maybe 50 requests per second, right? It's drowned out in big dashboards. But if you have precision tools, you can compare that to, you know, the combination of the build ID, the feature flag, you know, the the change you make compared to baseline.

00:31:38.210 — 00:31:38.770 · Speaker 4

Exactly.

00:31:38.810 — 00:31:46.050 · Charity Majors

Yeah, exactly. And then imagine that the change that you shipped was something more like a caching change.

00:31:46.090 — 00:31:46.490 · Speaker 3

Right.

00:31:46.530 — 00:31:50.610 · Charity Majors

Well, that might not surface for weeks months.

00:31:50.650 — 00:31:53.170 · Speaker 3

Yeah I agree. We can actually have.

00:31:53.170 — 00:31:59.610 · Charity Majors

Your agent check back up on it and be like, oh, this is a caching thing. We should check up on this for a long time and look for these things, you know?

00:31:59.680 — 00:32:00.120 · Speaker 4

Right.

00:32:00.160 — 00:32:00.680 · Speaker 3

Yeah.

00:32:00.720 — 00:32:01.360 · Charity Majors

I'm excited.

00:32:01.360 — 00:32:01.720 · Speaker 3

For that.

00:32:01.760 — 00:32:02.480 · Speaker 4

Yeah, I agree.

00:32:02.480 — 00:32:16.280 · David

I agree with you. And I think, uh, what you said is that that's the vision that excites me because I am bad at observability. Right? I'm, I'm. I don't know if you know the four temperaments. I am a sanguine. I don't think about risk. I think about jumping into this thing.

00:32:16.320 — 00:32:16.560 · Speaker 3

Right.

00:32:16.560 — 00:32:46.080 · David

So so I'm I'm that guy, so I don't I need a pessimistic person with me saying that this is what could go wrong. So from that point of view, when I see a lot of people developing stuff, what you said with what you're doing with honeycomb to look at with precision that, hey, this is what we'll do. We you have somebody sitting with you looking at what you're doing and looking at that impact of that small little change.

And it has the context too. So that's that's the cool part. That's what excites me about what you guys are building.

00:32:46.120 — 00:32:49.359 · Charity Majors

And instead of slowing you down, which I

00:32:50.680 — 00:33:03.580 · Charity Majors

acknowledge that if you're a software engineer using the standard tool stack, eating your vegetables could easily double, triple, quadruple the amount of time it takes you to do your job right. That's.

00:33:04.820 — 00:33:41.660 · Charity Majors

There are a few exceptional teams up there with the amount of engineering, discipline and leadership and that have never met. But for the most part, it's been out of reach for most people. And now I feel like it's finally becoming accessible to the median engineering team to bake these things in, to reduce the risk, to improve the precision so that, you know, honestly, I this is what made me finally get on AI train is just this excitement about this is our first chance to bring our engineering values to the mainstream.

00:33:41.780 — 00:33:49.020 · David

Yeah, no. I'm excited. I'm excited about all of that. And I think people are underestimating the role of observability in the future.

00:33:49.500 — 00:33:52.740 · Speaker 3

Like, without a doubt. Yeah, yeah, yeah. No.

00:33:53.300 — 00:33:55.580 · Charity Majors

Just replace observability with validation.

00:33:55.640 — 00:33:56.320 · Speaker 3

Yes.

00:33:56.960 — 00:33:58.800 · David

That's the outcome, right? Yeah.

00:33:58.920 — 00:34:23.520 · Charity Majors

Outcome engineering. By the way, my friend Corey Andre made this amazing site. Oh 16 G stands for Outcome Engineering. It is phenomenal. It's like the 60s. It's like Martin Luther. Uh, just, you know, stapling it to the church door. You know, this is it's about outcomes. And in order to track that intent, you have to be able to validate the outcome.

That's observability.

00:34:23.600 — 00:35:00.260 · David

Yeah. Yeah. Well, so as we like, I would love to understand some of the things that you can help in the listeners minds. Right. If what do you think are a few things that you have seen that people do as a pattern that you feel they should like start to change now with where things are going? Uh, you there's no there's no you can even shell and say honeycomb.

Go try this out. Look at this blog and things like that. Don't worry about that. But I think that'll be great for you to kind of help people understand, because in many ways, you're like the observability queen I'm talking to right now, so it will be great for us to kind of know and learn from you. So, yeah.

00:35:00.660 — 00:35:32.499 · Charity Majors

Oh. Thank you. That's very sweet. Um, I will I think I will show a little bit. Um, I do think that people, um. Yes. Obviously honeycomb um, is there's a, there's a second generation of observability tools. I, I called it observability 2.0 for a while, and I feel very embarrassed about that. It's like the most hackneyed, but like the wide structured events, you know, single source of truth instead of the three pillars.

Uh, it is so clear to me that the three pillars days are

00:35:33.660 — 00:35:46.379 · Charity Majors

numbered, no matter how many tens of millions of dollars of marketing and sales. But like, it just it just doesn't work. At a certain, you know, people should absolutely check out honeycomb. Um, I

00:35:48.260 — 00:36:05.330 · Charity Majors

if they aren't using open telemetry, adopt open Opentelemetry because it is vendor neutral. Um, it makes it so much. Will Larsen once said the sole scalable solution to technical debt is migrations. And

00:36:06.570 — 00:36:41.770 · Charity Majors

migrations are very hard if you've never done them or you don't do them a lot, but it's a muscle that you can build, you know? And when it comes to your observability tooling, you need to invest in the ability to, to, to because things are changing fast and you do not want to be stuck on the vendor that you were using 20 years ago.

Because if you are or even ten years ago, you're probably using the wrong thing. It's Gartner actually said in our last call. I don't think this is online anymore. Sadly, they were like, survivability is one of the only categories where there is no single dominant player for more than 20 years, ever.

00:36:41.810 — 00:36:46.490 · Speaker 3

Like, it always changes. It has to change with context, right?

00:36:46.530 — 00:36:47.010 · Charity Majors

Kind of makes.

00:36:47.010 — 00:36:47.610 · Speaker 3

Sense.

00:36:48.010 — 00:36:50.050 · Charity Majors

Hotel is kind of the the.

00:36:50.070 — 00:36:50.990 · Speaker 3

The the.

00:36:50.990 — 00:37:03.830 · Charity Majors

The lock in the door. I think that opens the ability to migrate, which is what gives you optionality, which is what gives you leverage in your budget conversations. Uh, the other thing that I will show is

00:37:05.830 — 00:37:06.910 · Charity Majors

the second I,

00:37:08.070 — 00:37:38.629 · Charity Majors

I appreciate the nice things you said about the first edition of the book. I'm not that proud of the first. I took us three and a half years, and there was no point that I was like, this is a great book, ship it. I feel it was like it was more like, it's been three years. I can't keep doing this. I can't I can't even look at it.

What if you just print it like I can't even read? You know, that was the vibe. Um, I feel differently about the second edition. I

00:37:39.630 — 00:38:02.689 · Charity Majors

we have some incredible guest authors. Um, some of the sharp. I mean, Matt Klein writing about mobile and front end Jeremy Morrell writing about instrumentation. Um, some of our direct competitor, you know, click House has a great chapter about how to use observable. You build a click out stack for observability.

Like I

00:38:04.250 — 00:39:02.440 · Charity Majors

my dog in this fight is not honeycomb. It is the old ways don't work. Y'all. You don't have to buy honeycomb, but you should be thinking about the sort of Ali 2.0 model. Um, anyway, and my particular section of the book, I got together with the coauthors, uh, last June, and I pitched them on. All right, let's add a new section to the book.

Part six. Uh, observability governance, written for observability engineering teams on the kinds of topics that they struggle with, uh, cost control and migrations and tooling and lots of stuff. And from June, July, August, September, I wrote, wrote, wrote. I was about done. And I'm like struggling with the last chapter on buying software.

And that's when I wrote a blog post and I'm like, hey, internet, I could use some help. I asked people to email me their stories about buying software or choosing tools, just in general. You know, and,

00:39:03.560 — 00:39:20.679 · Charity Majors

uh, the internet responded, I got it. I got many, many, many, many emails. And my progress in writing ground to a complete halt. And I

00:39:21.720 — 00:40:15.780 · Charity Majors

for over six weeks, I was just like staring at drafts. And a week before Thanksgiving I was up late, just kind of around with the emails. He was in cloud like water, this heat, and it hit me. Everything I had written was wrong. Probably useless. I mean, do observability teams need advice? Yes. Who does? Who?

The. Who the hell doesn't need help? Right? Of course they need help. But it wasn't going to do any good, because all these emails I was getting from people were describing a terminally buying process from the top down CEOs, VP's, SVP, distinguished engineers, principal engineers, directors, staff, engineers, observability teams.

No shared idea of what observability is should be, what problem is trying to solve, what problems should be trying to solve, whether you know and it just like.

00:40:15.820 — 00:40:16.460 · Speaker 3

No.

00:40:16.620 — 00:40:20.019 · Charity Majors

Shared context. And so like

00:40:21.060 — 00:40:24.100 · Charity Majors

just the stories of pain I, I

00:40:25.940 — 00:40:35.540 · Charity Majors

uh, an engineering org that spent six months doing all these POCs came to a very strong conclusion, sent the contract off, and

00:40:37.180 — 00:41:00.320 · Charity Majors

one of the loser's CEOs called up their CEO and he overruled them, didn't even talk to them and didn't consult with them, just like signed a different contract and again and and less dramatically. I mean, I think that inertia, just like it's paralyzing, so much has changed and it's become so. Do you realize most people

00:41:01.640 — 00:41:38.000 · Charity Majors

don't realize this? But observability is now typically the second highest costs behind cloud spend when it comes to tools or spending on software, there's cloud and then there's observability. And observability is usually 20 to 30% of the overall infrastructure bill. It's astronomical. Right. And yet 15 years ago it was 5050 K.

It was it was a it was an ops tool cost center rolled up to the CIO.

00:41:39.960 — 00:42:01.860 · Charity Majors

So things have changed right? Okay. So I get all these emails. I realize what I've written is not actually helpful. And and right before Thanksgiving, I kind of had my Saul on the road to Damascus sort of conversion moment. And so I or less three and a half months, I, I, I threw everything out and I rewrote it from scratch.

And now

00:42:03.260 — 00:42:20.820 · Charity Majors

my section of the book is still called Observability governance. But it opens it's, it's, it's for technical decision makers from the top down. Right. So it starts with like an open letter to CTOs being like observability is not so 2019. So pre AI it's actually

00:42:22.140 — 00:42:25.420 · Charity Majors

it's how your team your organization learns and.

00:42:25.420 — 00:42:26.020 · Speaker 3

All.

00:42:26.020 — 00:42:33.940 · Charity Majors

Of your AI initiatives are backed up behind. You know the the blockers is no longer writing code.

00:42:33.940 — 00:42:36.900 · Speaker 3

It's understanding exactly from it.

00:42:36.940 — 00:42:38.980 · David

Yeah that's pretty prudent actually.

00:42:39.170 — 00:43:33.010 · Charity Majors

Yeah. So we start with the open and then because every tour, every technical term is so overloaded with sales and marketing bullshit. So, um, the next couple of chapters, we talk about, um, we just use systems thinking terms. Right? Feedback loops. Feedback loops can be amplifying. That's what causes change.

Or they can be balancing, which is what causes, you know, stability. But observability is the thing that connects cause and effect to create a loop. No observability, no loop, no feedback loop. And if that and if that observability is, uh, laggy lossy delay, uh, then you're balancing loops. Instead of causing stability, they either freeze up or they oscillate.

00:43:33.250 — 00:43:35.050 · David

Absolutely. Yeah. Yeah, absolutely.

00:43:35.890 — 00:43:41.150 · Charity Majors

Without observability, they're not positive loops. They're doom spirals.

00:43:41.590 — 00:43:57.630 · David

No, what I was going to say was I kind of. I never thought about it like this, but when you said an open letter to the CEOs, I think it's very important for this kind of idea to rise up in the minds of the, you know, not just leadership.

00:43:58.030 — 00:43:59.070 · Speaker 3

Yeah, yeah.

00:43:59.550 — 00:44:11.870 · David

Yeah. Because this is tied to the outcomes that you're driving, right? It's tied to your customer experience. It's tied to the customer experience. And then drive revenue. So people don't think like that. But it's good that you.

00:44:11.910 — 00:44:12.870 · Speaker 3

Did that ship.

00:44:13.230 — 00:45:19.930 · Charity Majors

Exactly. There is this perception. One of the problems with technical executives is that it's really hard to outgrow the things that you learn when you are hands on. You know, I feel this myself all the time. It's like the that when I was in the thick of it at parse, right, ten, 15 years ago, I know that so deeply.

And the stuff that I haven't been hands on with for the past few years, I know a lot less deeply. Right? And most CTOs and when they were hands on, uh, they had monitoring and logging and it was an ops tool and it was a cost center, and it was related to, uh, reliability and outages and incidents, but which is important stuff is important.

But the developer feedback loop is about building and creating value and learning. Like these are two different. The ops feedback loop and the developer feedback loop are two different things. And observability for developers is right in the critical path of building and understanding and serving your users.

00:45:19.970 — 00:45:31.490 · David

Yeah, yeah. No, absolutely. And well said actually. And these are like I think we don't think about these things enough because we are just so focused on let me build my feature okay. Now this.

00:45:31.490 — 00:45:32.170 · Speaker 3

Feature needs to.

00:45:32.170 — 00:45:33.650 · David

Scale. Yeah.

00:45:33.720 — 00:45:34.840 · Charity Majors

Everyone's overloaded.

00:45:34.880 — 00:45:35.800 · Speaker 3

Right. And so.

00:45:36.040 — 00:45:42.360 · Charity Majors

Yeah, I tried to make it as simple as I possibly could. Just simple and intuitive and vendor neutral. So that's what I'm shilling.

00:45:42.480 — 00:45:42.680 · Speaker 3

Yeah.

00:45:42.720 — 00:46:35.060 · David

No, it's very good. I mean, it's it's it's been it's awesome. Like, you know, for everyone listening in. So the two things that you want. You go check out honeycomb. Obviously some of the cool things that chat is I believe you have a pretty active Twitter and Substack too. So go go check that out. We'll put that in, um, with the with the post that we do.

And obviously, uh, go check out the Observability engineering part one if you want to. But according to charity, part two is much more cooler. So go check that out. Uh, yeah. It's been an absolute pleasure having you on charity. I mean, um, as we leave, like, I. Now, let's just talk about. I think you brought this up, and I always love talking to veterans in the industry.

Uh, it's sort of like. What's your advice to junior engineers, right? Like, uh, it'll be good for you to get, uh, spray some of those thoughts out as well. And for those young people who are listening from people like Charity, like, what is it that they should be focused on learning and what should be their mental model?

00:46:35.540 — 00:46:55.980 · Charity Majors

I really struggle with how to give advice to young people because the world is so different. When I was, you know, 1722. I honestly feel like the most useful and relevant advice rarely comes from people who are this far ahead of you. It's usually from people who are in your shoes. Five years ago.

00:46:56.140 — 00:46:56.860 · Speaker 3

Makes sense.

00:46:56.860 — 00:48:23.200 · Charity Majors

And I you know, Kennebec has written some amazing things about how junior engineers are going to be more AI natives than we are, right. But to get those skills, it depends on people being willing to hire them. So instead of me trying to give any advice to junior engineers and screwing it up, I'm just going to send them empathy and good vibes and hopes.

And instead I'm going to say a different thing, which is every company I know of that is invested into hiring training junior engineers. Over the past couple of years, has number one not regretted it? They have been very grateful, like vocally happy about it. And number two, it has not come from the top down.

It has been championed bottom up by other engineers, by senior engineers, staff engineers, usually not even managers, but like senior engineers and staff engineers who are like, we are cannibalizing our future by not investing in the next generation. I will personally take on the responsibility of mentoring these kids, interviewing them.

You know, give me give me some, you know, so I would say if you're a senior IC, don't feel like you have no power because you have power and you might be the only one who will make this happen.

00:48:23.480 — 00:48:23.920 · Speaker 3

Yeah.

00:48:23.960 — 00:48:54.620 · David

Very cool. Well. Well said. No, I, I have more positive. Uh, you know, perspective, like. Similar to what you said. Like, I think it's more incumbent on the veterans to kind of look at the way the tech is going to shape the world and always remember that it's we all learn from somebody blogs, we all learn from somebody StackOverflow and we all learn from places like that.

And and that's the kind of culture that we experience. So we should kind of remember that in as we progress.

00:48:54.940 — 00:48:56.900 · Speaker 3

I would echo that it is.

00:48:57.100 — 00:49:31.330 · Charity Majors

Once you've seen what good looks like, it seems obvious. Yes, obvious and intuitive. It's obvious that if you ship code in 15 minutes, it's obviously this is easier, right? Uh, it's not obvious to people who haven't seen it, and it often looks harder and out of reach. Right? So I think humility is called for.

I think empathy is called for. And I think just trying to reach people where they're at instead of asking them to try and reach you where you're at.

00:49:31.370 — 00:49:32.090 · David

Well said.

00:49:32.130 — 00:49:32.770 · Speaker 3

Awesome.

00:49:32.810 — 00:49:58.610 · David

Well, thank you again, Charity, for coming on. It was such a pleasure talking to you about so many different things, especially like I really loved your perspective on like the way AI observability is kind of shaped the world, and also how executives need to start thinking about this, not as an afterthought.

So I really appreciate that. And for all the listeners, go check out charity, her company, honeycomb. Um, and the book that they it's coming out. Is it already out or is it coming.

00:49:58.610 — 00:49:59.130 · Speaker 3

Out in.

00:49:59.170 — 00:50:07.730 · Charity Majors

The dead Tree version will be out in June. Uh, chapters preview chapters are coming out in O'Reilly and the honeycomb blog between now and then.

00:50:07.770 — 00:50:18.450 · David

Very cool. Yeah. So check it out and we'll post all the links, uh, in the part in the podcast description as well. So thank you everyone for joining us once again. Appreciate it. And Chidi, thank you once again.

00:50:18.490 — 00:50:19.210 · Speaker 3

Thanks.