Things Sam Says on Podcasts

See, I feel like I've got to channel my friend Charity at this point and say the truth is production. Everything else is the lies we tell ourselves. Production is truth. But I think in terms of like a definition of what the system should be doing, there is this ongoing debate about where is that definition? Is it the code or is it the spec? Brigitte Broccoli did a nice article on Martin Fader's website recently about spectrum development. And she talks about the idea of some companies using being spec first or spec anchored. And a lot of it comes down to where they see the source of truth being. So a lot of companies are using things like Kiro or SpecKit to define a feature and then the LLM executes that spec and you can iterate on that process, right? And so a lot of people, they start off, they use the spec to write the code and then they throw the spec away. And then you get other companies that keep the spec and then when the feature changes, they just update the spec. And so they're sort of seeing a bit, the code and a bit, the spec as a source of truth in terms of what the system should do. And then you're now starting to get companies where they don't even ever look at the code and they just look at the spec. And so we're kind of going on this spectrum. She had some great terms for these. And I think for me, it's like, is that the right path? I don't know. I think there's a lot of questions about what we would need to do to get that. We have a degree of confidence about what happens when I execute a line of Java code beyond the safety of the compiler and the runtime and all that stuff. There's a lot more questions to answer, I think, before the whole world is ready to say the spec is the source of truth of what our system is supposed to do. Production is always going to be the real truth, though, just to be clear. But

I think it's a good question to just ask yourself, however you're building and however you're using agents, of where is the truth of your system right now? Is it more in the code? Do you have a spec? Is it scattered between the minds of people? Which goes back to the collaboration. Because again, a lot of the reality of a lot of startups are is a few people hold these things in their head, right? Which is okay in the beginning. But of course, as you scale, one of the reasons we used to love code review, which is now going out of fashion, but we used to share information. And when someone left the company or got ill or the famous bus factor, it was okay because other people understood because they reviewed they were taking part in that part of it. But now we're, you know, with AI code review is some companies are like, well, we're not going to do it because of it's with too much code. That

Which also goes back a little bit to what Charity was talking about on how you want to have systems that value that works about having empathy, for example,

for the work that QAs and SRE folks have had to own code in production that they did not write. They don't necessarily read or even understand, but they know how to operate. They know what good looks like in production. They know what it looks like when it goes poorly.

I mean, that's also a bigger thing that now more developers are having to struggle with, because if you want to do spec-driven development, you have to know what good looks like. I run these sort of half-day conferences for O'Reilly online. We did one recently on software delivery. A load of speakers and about six speakers, and four of them were talking specifically around different varieties of spec-driven development effectively. And all of them came back and says, you can't do this unless you can define what good looks like. And if you can't, don't bother. And, you know, some of those people were much more bullish, like Addy Osama is very bullish on all of this stuff. But one of the presenters who was doing, they've got like a small software fraction, only team or three people, but like they're actively pulling in their observability data, their OTEL stuff, right, into their agent to understand how it's changing. And I think that's exposing, I think, more developers into understanding. I mean, this comes back to the acceptance criteria, stuff people do in stories. They were always very narrow around functional requirement stuff. So now it's like, I think we'd have to be more grown up about those things. The discipline is absolutely important and the expertise is still needed. It's just needed at a different place.

like, I think we're talking about like solving it with when you have same resources, but then we get into things like scaling. Like, can you scale horizontally, add more resources? Because there's two sides of this, right? Like whenever you have it, and so what point can you do that? Or are you going to face a bottleneck because you still have a central whatever? But

like that's

Yep.

And you still can't keep up. So that's a perfectly valid mechanism for some things, but you kind of have to know the context in which to say you can't just scale yourself out of trouble.

I think we're going back to a lot of times as like as an engineer, you want to know about the business. You want to know what creates a value, what creates revenue, what is important for the company versus what is not. And then where we want to maybe invest to with our infrastructure, with our reliability, or is this where it doesn't matter and it's not worth doing? It's

I mean, there was always, I mean, also in those sorts of environments. I mean, I was checking people at SoundCloud that's similar about that. And this is also like about a corporate culture and like what was acceptable. Like in that kind of environment, just go and do stuff. It's absolutely fine.

And I think that's why Uber, for example, went in. I just talked with Twan Palm, the C2 at the time, about like what was the real deal about microservices. And he said, Uber, like, he never really wanted that many services, but it was just more of a constraint because Uber had a monolith. The story was like, which he told on a podcast, we'll link it at the show notes below that Uber had this big monolith and it was really slow. It was slowing delivery down. And they needed to move fast because all the other competitors were out there. So they said, all right, like they wanted to get rid of the monolith and put together smaller services, but they knew unless they had a strong mandate that you cannot add new stuff to it, this will never happen. So they said the mandate was everything new needs to be your own service. You take care of it. We don't care. And suddenly so many independent services were born, which was not the goal. And years after the fact that when Uber decomposed the monolith, they're now looking at maybe sense to be merging based on business domain, et cetera. But this story wasn't really told at the time. It was just, oh, look, like Uber has thousands of microservices, almost one or two per engineer, right? And this is what the world saw.

Really?

Yeah, yeah, absolutely. Because it pushes you quite naturally to a more aggressive distribution of your state and everything else. And so you have to have a good reason. One of the best reasons to use microservices is if what you actually want is an organization with a high degree of autonomy. If what you want is teams who are able to act independently from each other, the microservice architecture is a fantastically good fit. But then really it's a mean. To an end, right? So that's often what a lot of people get wrong, I think, is they adopt microservices and they think some other magic stuff's going to happen. If you want organizational autonomy, that's what you're working towards. And this is just one of the mechanisms to get you there. And most of the enterprise organizations, the vast majority of ones I've worked with in this space, that's what they're after. They've got huge numbers of developers, lots of middle management. They're not getting much work done. They're thinking, what do we do? Organizational autonomy becomes, you know, the solution to all problems. Again, I don't always get to have to say whether or not it's a good idea. And I think that's where microservices can be especially useful. I mean, there are other exceptions as well. And so an organization like Uber, I mean, and I say, I know at Netflix as well, things like Domain Dream Design weren't talked about at Netflix, but the teams were already fairly well scoped around areas of product already. And so you had those natural high-level boundaries around the business domain. So those things flowed naturally. And so that for me was a logical kind of stepping stone. You also have the difference, and this is something I realized quite early on. When you go and do the work in the dot-coms versus the enterprise organizations, when you

say dot-coms, you mean the digital companies, tech companies.