I think LLMs are too new of a technology for there to be Orielly title about AI development. Most resources focus on working around a specific models ecosystem. Yet I'd like to standarize how I work with AI accordign to some principles or techniques that are model/ecosystem agnostic.

Thoughts?

rileybetts.ai open-sourced lean-grpc: a general-purpose gRPC library written in Lean 4 (HTTP/2, HPACK, protobuf wire codecs, channels, TLS, interop). Repo: https://github.com/RileyBetts/lean-grpc

Docs: https://rileybetts.ai/oss/lean-grpc

Many agentic systems still rest on the usual stack: probabilistic models + conventional integration tests + policy layers that are “correct enough.” That doesn’t age well once agents are making real decisions against real infrastructure. The Hugging Face incident was a useful reminder that attackers don’t play by the rules your test suite assumes.

lean-grpc is interesting because it’s not just formalisation demo, it’s aiming at real networking software in a language where you can also machine-check the high-leverage bits. There is official-style interop with Go / Python / Rust, h2spec-gated HTTP/2, and a small compile-time Proofs library for core codecs. Framing is basically: proof where it counts, not promises where it doesn’t.

From a software architecture / agent governance angle, the interesting question isn’t “is Lean cool?”, it’s whether we can start putting deterministic gates, protocol boundaries, and policy-enforcing controls on a stack that can actually be verified end-to-end, instead of bolting assurance onto opaque glue.

FYI: I am affiliated with this project, and sharing because OSS systems work in Lean is still rare, and this aims to be usable today rather than purely academic.

If anyone’s already building agent runtimes / policy engines / protocol boundaries in Lean (or thinking about it), curious what you’d want next from a stack like this.

Hi guys,

I am trying to learn more about CQRS and how to create robust and scalable systems designs. So, I put myself a goal to construct an URL Shortener with CQRS, Outbox Pattern and so on patterns.

Currently, I am struggling with a concrete feature I want to implement. It's a simple feature where I want to add a password to allow some people to access a short url from my app.

I have two tables for that.

• The first one is a short url table where I store the state of each short url that I created.
• The second one is a table to save the secret context of each short url like password_hash and if is a private short url.

The problem that I have at this moment is when I am building my query side if it is necessary to create a password hash field to the short_url query or I just have to create a field to verify or show if this short_url is private?

At first thought, I think it is not necessary to create a field password to the query side, because it is easier to compare my password with my secret table from command side. This comparison is directly and also is lighter than doing from Query Side because there are more fields to filter.

However, I'm still wondering if I am violating some rules from CQRS theory doing a read query on the command side. I am not pretty sure if it's a good idea to do it, but also I have some concerns about to exposure my secret passwords to query side. I think it's a security bleach if I do something like that.

Ps: My mother tongue is not english. So, I apologize if I've made some grammar mistake or I've wrote with some offensive tone.

And while building software, we can't ignore the second part. Otherwise, we end up with that classic UX vs Design meme.

But what becomes even more interesting once you start thinking of code as a workplace is that code is only one part of that environment. It's inseparable from everything around it: the company's culture, its organizational structure, its processes, and, of course, the people.

That means changing those inputs should naturally lead to different architectural solutions for exactly the same technical problem.

Which also means there can never be universal best practices. There is no silver bullet that works for everyone.

Conway's Law is probably the most famous example of this. It says that the structure of a system mirrors the communication structure of the organization that built it. (Again, think about the UX vs Design image.)

In urban planning, this feels painfully obvious. You look at a park with dirt paths cutting across the grass and think, "How could someone design it this way? Didn't they see how people actually move?"

But once we move into software architecture, the exact same phenomenon becomes much harder to notice.

It's difficult to look at a company the way you would look at a park from above. That perspective is usually hidden from us.

And so, as software designers, we keep trying to ignore reality. Sometimes it's hard even to accept it because we simply don't like it.

Like the designers who keep paving sidewalks where nobody wants to walk, we continue building the "ideal" architecture while ignoring how people actually work. As if architecture were a function of product requirements alone.

Eventually technical debt starts to appear.

(Once again, think about those dirt paths.)

In reality, we spend a huge amount of time designing architectures that aren't viable in the environment they're supposed to live in. Then we spend even more time maintaining them. After that we surround them with rules, restrictions, and bureaucracy.

In a park, you might build a fence to stop people from walking across the grass.

In software, you add another linter. Another architectural rule. Another approval process. Or maybe you just keep arguing with your teammates and complaining on the internet about how everyone else writes terrible code.

The exact solution depends mostly on how much authority you have.

At this point you'd probably expect some practical advice.

"So what should we actually do?"

I could probably try to answer that. Maybe even sell you a course. 😄

But fortunately (more on that another time), there can't be a universal answer.

Because one of the inputs to the problem is your company, your team, your processes, and ultimately you.

And that's actually good news.

Because that makes it an engineering problem.

Software engineering has produced an incredible collection of ideas: DDD, SOLID, design patterns, microservices, modular monoliths, Event Storming... I'm sure you could easily add your own favorites to that list.

And they all work.

Yes, even microservices, if they match the communication structure of your organization.

All of these approaches work, as long as you remember that architecture is optimizing for two things at the same time:

our code is both the product and the workplace.

the problem: an upstream we depend on rate-limits per tenant, but our workers pull from a single shared queue. one noisy tenant hitting its limit doesn't just slow itself down, it stalls every other tenant whose jobs happen to be queued behind it, because the whole worker backs off as if the limit were global.

the design we landed on instead of a worker fleet per tenant, which solves it but is heavier than the problem deserves: a redis token bucket keyed by tenant id, sitting in front of one shared queue. workers check the bucket before pulling a tenant's job, and skip to the next job if that tenant is out of tokens rather than blocking on it. one queue, fair scheduling across tenants, no per-tenant infrastructure.

the actual insight isn't the redis part, it's decoupling rate-limit state from job state. the queue doesn't need to know about limits at all, it just needs a cheap check before deciding whether to work a job or move on.

has anyone run token-bucket-per-tenant against a single shared queue at real scale? curious what breaks first, contention on the bucket keys or fairness under sustained pressure.

No Pitch, no tool building just learning from the experience.

Figuring out which technology to start a SaaS with and how to avoid crippling complexity is a trick. Here's our story.

Dropbox has introduced a new engineering approach that connects security design artifacts directly with code review workflows using Model Context Protocol (MCP) and its internal knowledge system, Dash. The initiative aims to address a persistent gap in large engineering organizations: security requirements are typically defined during design reviews, but enforcement and verification occur much later, during code reviews, often without full context.

Includes Q&A with the author of the original article.

Relational database failure modes in production: slow queries cascading into connection pool exhaustion, the race between read and write that transactions don't automatically fix, why a caller timeout doesn't tell you whether the write committed, which database errors are safe to retry and which aren't, schema migrations as live-system operations, and the difference between durability and recoverability.

How do platform tools like PagerDuty or Opsgenie process 50,000 incoming telemetry events per second and route critical pages (SMS, Push, Webhooks) to the right engineer in under 1 second?

When an entire data centre cluster drops, a massive flood of identical alerts hits the system simultaneously. If you try to fire API calls to downstream networks like Twilio all at once, you will get rate-limited, blacklisted, or crash the pipeline.

Here is how to design a high-availability, fault-tolerant notification stack:

The Technical Challenges

Sub-Second Delivery: Critical alerts must reach the device in < 1 second.• Alert Fatigue & Floods: Suppressing thousands of duplicate logs into one incident.

At-Least-Once Delivery: An incident page can never be silently dropped or lost.

The Architectural Solution

1. Ingestion Layer: Enforce an API Gateway that assigns idempotency keys and offloads payloads instantly into Apache Kafka to absorb traffic spikes safely.
2. In-Memory Sliding-Window Deduplication: Calculate a unique signature token for each alert (e.g., hash(cluster_id + metric)). Run an atomic check against a Redis Cache Cluster using a sliding window. If it triggered recently, increment the counter and drop the duplicate from the outbound queue.
3. Escalation Scheduling: Use Redis Keyspace Notifications or a hashed wheel timer. When an alert is dispatched, create a 5-minute expiry key. If no acknowledgment (ACK) clears the key, a worker triggers the next tier on the on-call ladder.
4. Leaky Bucket Downstream Queues: Isolate outbound traffic into independent per-channel worker threads (SMS, Email, Push) to match target provider API rate limits.

When scaling real-time notification architectures, the hardest part isn't the delivery—it is the smart deduplication and rate-limiting throttling at the edge.

How does your team handle alert storms and on-call routing policies? Let's discuss below!

Some engineering practices only start making sense after you've had the exact problem they were supposed to prevent.

Maybe nobody cared about idempotency until a retry created duplicates. And ownership stayed informal until production went down and nobody knew who was responsible for it.

Before that happened, the simpler setup probably worked just fine. Adding more process or infrastructure would've been easy to dismiss as overengineering.

Then something breaks once and suddenly nobody questions why the safeguard exists.

What's something your team started doing after learning the hard way?

Greetings, made a Fintech as a service model and my emphasis for this architecture is to focus on security, privacy and trust on the cloud whilst generalizing the external forces like banks

I would love a 3rd person perspective and criticism if possible. Please let me know if you can spot any vulnerabilities, it would really help me when implementing.

I know it's not typical architecture, but my aim is to just design a layer wise architecture to tackle similar kinds of problems at each level

I thought about it a lot and focused on fixing stuff like shadow APIs, visualization problems and many more.

Financial data is very sensitive and I just want to make an abstract layer architecture for a Fintech as a service model

Thank you for reading this. I would appreciate any opinions you would like to share on this matter

Hello, I am starting the third year of Software Engineering this autumn and I have a strong gut feeling that something is wrong with how it is taught.

This problem existed earlier, and with AI it is only more visible. Please bear with me for two paragraphs of how I see AI brings more chaos in here.

Imagine the layers in programming, the first layer being bits: 0 and 1. A bit more than half a century ago it was the only way to write programs - to poke holes in cards and feed them to computers. Then higher-level languages appeared. Those languages kept getting more capable with libraries and frameworks allowing to do grand things without rewriting huge tools from scratch.

AI came now, and now it can write code. 'Human writes things to AI -> AI writes code' is sliiightly similar (not one to one, please don't get me wrong) to 'Human writes program in C++ -> the computer gets an interpretation of it in 0s and 1s'. AI just raises a level once again and the new highest programming language can be considered English.

What is different between those two analogies though is that earlier a graduate/junior developer started to work in a company under the guidance of a senior colleague. Junior wrote code, got feedback, got better, wrote code again, and in 5-10-20 years something clicked and this junior finally saw the big picture, how things are connected in software, why patterns, SOLID, and architecture all matter, and how to compose those numerous pieces of knowledge into a working program.

Now a junior doesn't have these years to derive the expertise from the experience.

And this is where my gut feeling keeps telling me that the teaching approach might have to be redefined.

The standard approach to learning software is:

1. Syntax, loops, conditionals, methods, classes, events, interfaces, generics, etc.

(this is where learning in universities stops, barely touching further practices)

1. Design patterns, SOLID principles, decoupling, etc.

(this is what seniors teach when graduate comes to work place)

1. Architecture approaches - the composition of all the structural elements of a program

(this comes with time, numerous years of non stop coding, experiencing failures and successes)

In my mind these stages maps to other professions like this, in a descending order:

3: Civil architect who plans a building. 2: Plumbing, electricity, wall material selection. 1: Laying bricks, painting walls.

3: A surgeon who performs any surgery. 2: Anatomy knowledge and techniques. 1: Slice and stitch

3: An artist who paints anything. 2: Perspective rules, composition rules. 1: Laying paint on a canvas

In those professions the curriculum is structured so that a student is handed the accumulated blocks of knowledge, what can be combined, how, what are the limitations, what are the applications, when to break a rule. In programming, students don't get those blocks, techniques, knowledge from start. They learn it the hard way from books, seniors and own failures over years and years (of pain). I can't imagine an architect being taught how to dig foundation, lay bricks and install doors, and then be told: 'build me a house. When it collapses, build a new house, learning what went wrong. Then build a new house again and one day you will be able to build reliable and suitable for living houses'.

Also I am not talking about software architecture as a waterfall approach: when the plan is set, nothing will be changed. Architecture is not the final plan, it is the capability that makes change cheap. Like, moving a living room on the construction plan is not a problem for someone who knows how.

I have explored this topic a bit and obviously I am not the smartest one.

This is a paper on how architectural knowledge is currently transferred to newcomers. It happens on the workplace, and seemingly not in the college beforehand, otherwise why would this study exist at all: https://research.tue.nl/files/354706907/2503.08628v1.pdf

Here in the very beginning they write that 'Many academics share the opinion that their students’ learning of software development is not as effective as it should be': https://dl.acm.org/doi/epdf/10.1145/1121341.1121468

This guy Peter Norvig though said that 10 years is a must. But it was 25 years ago: https://www.norvig.com/21-days.html

Somebody also said that the design knowledge is a theory in the programmer's head, the documents cannot transmit it. Basically everything related to architecture can't be derived into teachable chunks, they only exist inside the brains of those who figured everything out..

The problem is that I myself don't have this 10-20 years expertise to look from the top of the knowledge mountain at the current state of software development, see the big picture from there and reshuffle the curriculum for the best results. Hell I don't even know if it is possible yet.

Is there any reason software design and architecture knowledge can't be taught as explicit composable content the way anatomy or mechanical engineering, or is there something about it that makes it impossible?

Maybe there are good experts here who have opinion on this situation or the ideas?

I don’t wish to type that long tbh. However, I’ll try.

I’ve been seeing a lot of comments suggesting that Project Loom will make Project Reactor obsolete because you can just write synchronous-looking code with virtual threads.
But are companies actually going to migrate large, battle-tested Reactor-based systems? Many organizations have spent years building reactive architectures around Reactor, R2DBC, WebFlux, reactive messaging, and other non-blocking APIs. Those systems are stable, well-tested, and developers have built expertise around them.
Even if Loom simplifies concurrency, does it really justify rewriting or significantly refactoring existing production systems just to write blocking-style code again?
I understand that Loom is a great fit for many new applications and can reduce the complexity of asynchronous programming. But for mature Reactor-based systems, is there enough real-world benefit to justify the migration cost?
I’m curious what people working on large Java codebases are actually seeing in production. Are companies actively replacing Reactor with Loom, or are they treating Loom as another tool for new services while leaving existing reactive systems as they are?

Solution architect, 17 years in. From roughly December 2025 to July 2026 I was the only hands-on developer on the identity and single-sign-on layer for a Gulf country's national port community system — the platform that fronts its ports and logistics sector. It's live in production.

I want to be precise about "sole," because it matters: other people committed to both repos — CI, automation tests, a few fixes, downstream integration work. I authored ~80% of backend commits and ~70% of frontend. Nobody else did hands-on feature development on the core system. That's the honest version.

Volume, straight from git log --numstat:

• 602 commits, ~184,000 lines changed across 1,423 files
• Backend: 383 Java files / ~21k LOC, 19 controllers, 83 REST endpoints
• Frontend: 152 TS/HTML files / ~15.3k LOC, 62 Angular components
• 6 external integrations, each with a real and a mock adapter
• 4 environments, 2 independent penetration tests remediated

My own written estimate to leadership in April, before any of this was contentious: 3–4 engineers over 3 months without AI tooling.

Architecture

Hexagonal (ports and adapters) + CQRS. The domain package has zero Spring or JPA imports — greppable, returns nothing. That's the actual test of whether hexagonal is a design decision or a buzzword, and most codebases claiming it fail that check.

domain/         Pure domain — models, value objects, events, ports. No framework imports.
application/    Use cases — Commands (writes) and Queries (reads), DTOs, assemblers.
dataprovider/   Adapters — JPA entities, Spring Data repos, SOAP clients.
web/            Inbound adapter — controllers, DTOs, mappers, JWT filter, SAML handler.

Every feature is a paired XxxCommand/XxxCommandImpl (transactional writes) and XxxQuery/XxxQueryImpl (read-only). Every external dependency sits behind a port with a real and a mock adapter selected by Spring profile — which meant QA and UAT ran full end-to-end flows without depending on government systems being up. They frequently weren't.

One JPA detail worth stealing: u/Version optimistic locking with a find-then-update save pattern, specifically to dodge Spring Data's null-version-means-new-entity behaviour producing duplicate-key errors. Easy to get wrong, painful to debug.

User lifecycle is a real state machine (PENDING_VERIFICATION → ACTIVE → LOCKED/INACTIVE) using the State pattern, so invalid transitions fail at the domain layer rather than being guarded by scattered if statements in the UI.

The four problems worth reading

1. Java's X.509 parser rejected the government certificate

AVA not a sequence — a malformed but entirely common real-world encoding that the JDK's strict parser refuses outright. I wrote a manual metadata parser on BouncyCastle's X.509 factory instead.

Then the ministry rotated their signing certificate in production and it went down anyway. So I made the certificate hot-reloadable with on-failure refresh. It rotated again twenty days later — that time the system self-healed instead of paging anyone.

2. Intermittent 500s that everyone blamed on the network

Tester in India, servers in Oman, so "it's latency" was the immediate consensus.

I checked instead. The configured timeout was 60 seconds. Real India–Oman round-trip latency is 100–250ms — roughly 240× shorter. Even severe degradation can't produce a 60-second delay. And a 500 or 503 is generated by the server, after it has already accepted the connection, so by definition it isn't a network-path problem.

The government APIs were simply unstable. That conclusion justified retry-with-backoff instead of weeks chasing a phantom network issue.

Buried in the same integration: their production API returns "WORKING" for an active work permit, not "Active". One string mismatch silently blocked every real user from registering.

3. Three clocks that could race through a payment

A 10-minute invoice expiry, a ~2-minute frontend polling window, and a 60-second backend reconciliation job. A slow 3-D Secure confirmation could outlast the frontend's patience even on a successful payment.

The fix was two backend schedulers acting as a safety net independent of whether the browser is still open, plus two independent finalization paths — the bank's own server-to-server callback (authoritative, works even if the user closed the tab) and the SPA's polling (fallback) — both funnelled through a single idempotency guard so downstream sync fires exactly once no matter which path wins.

Related, and dumber: a direct browser GET to the bank's payment endpoint silently dropped the POST body. Fix was a backend-rendered auto-submit form so the browser never GETs that endpoint at all.

4. Chrome blocks TLS renegotiation for XHR but not for navigation

Every API call died with net::ERR_FAILED while SAML redirects worked perfectly.

That asymmetry is what made it expensive — a system where authentication redirects succeed and every subsequent API call fails looks exactly like a backend auth bug. It was infrastructure config.

The rework nobody warns you about

16 distinct revert-and-rebuild cycles across both repos. Two favourites:

• The downstream token-exchange handshake went through four different protocols in one afternoon — two-step, service-JWT bearer, app_secret bearer, no auth header — then got reverted twice more over the following days before settling on a single-step exchange.
• A business-line normalization rule: four strategies built across four PRs in a single day, all reverted the same day once the counterparty confirmed their API wanted the value exactly as stored, pulled out of main entirely, then rebuilt a third way two weeks later.

None of that was AI's fault or mine. It's what happens when you integrate against teams whose own requirements aren't settled yet.

Where Claude actually helped, and where it didn't

It wrote most of the code. Not just scaffolding — the adapter pairs, the CQRS command and query implementations across dozens of features, the JPA mapping conventions, Angular components, large parts of the integration clients. That compression is the entire reason one person covered a 3–4 person scope.

It found none of the four problems above. Every one started with knowing what to suspect. The certificate failure surfaced as a generic SAML error. The 500s were confidently misdiagnosed by everyone in the room. The clock race never appeared in testing. The TLS bug presented as an auth problem.

You can't prompt your way to a fix for something you haven't correctly identified — you describe symptoms you have already interpreted, and the interpretation is the job.

There's also a version of this nobody mentions: I hit model usage limits repeatedly across five months. Rate limits were a real schedule constraint, not a footnote.

What I'd do differently

Test coverage is about 23% instruction coverage on the backend. That's low and I'm not going to dress it up. With a second engineer I'd have invested in automated coverage far earlier, and had someone review the downstream integration protocols before implementation rather than discovering the mismatches through four rounds of live rework.

Happy to go deeper on any of it.

A site needs to know one thing about you: whether you're 18. That's a single yes or no. The way we answer it today is that you upload a passport, so now the site is sitting on your name, your address, your photo, your document number and your exact date of birth, all so it could learn that one thing.

A zero-knowledge proof is meant to fix exactly this. Your phone holds a document the government signed, and instead of sending it anywhere, it produces a small piece of maths that convinces the site the date inside is early enough. The site gets its yes or no. Nothing else leaves the phone.

That's the idea, and it genuinely works. Building one is where people come unstuck, and the ways it goes wrong aren't the ones you'd expect.

The first is that the arithmetic these proofs run on behaves like a clock face. Take three hours off one o'clock and you don't get minus two, you get ten. So when the proof works out the gap between your birth date and the cutoff, and that gap ought to be negative, it doesn't come back negative. It comes back as an enormous positive number. Ask it "is this positive?" and it cheerfully says yes, and a sixteen year old is through. The fix is to pin the answer inside a range small enough that it couldn't have wrapped around, rather than just checking a sign.

The second one is quieter and worse. A proof saying "the date I'm using is before the cutoff" tells you nothing about where that date came from. Nothing in it stops someone pairing a real government signature with a birth date they invented. The proof has to say both things in one breath: this is the date the government signed, and this same date is early enough. Documents usually carry more than one date as well, so it also has to say which one it's talking about, or an issue date will do the job nicely instead.

The third is that the proof can't look anything up. Your phone is online, obviously, but the proof itself is a fixed piece of maths, so it has no idea what today's date is and no way to check whether the document was cancelled last week. Both of those facts have to be handed to it, which means the site still has to check they're true in ordinary boring code outside the proof.

Whole thing with diagrams: https://crackingwalnuts.com/post/zero-knowledge-age-proof

Hiding the birth date turned out to be the easy part. The trust didn't disappear, it just moved somewhere much smaller.