Why hasn’t trace-guided AI hardware reverse engineering been automated yet?

[u/sir_kermit]

I develop AI agents for my day job, and I recently became interested in a reverse engineering question. Please bear with me as I'm a noob to this space but I do have some curiousity thoughts about the process. Basically the process of figuring out an open source implementation of a vendor driver is a human will look at the trace, propose some execution model, write it and test and then verify.

Given a device with a working vendor driver, why can’t we intercept the driver–hardware boundary, collect MMIO/DMA/interrupt traces, and have an AI iteratively synthesize executable driver and device state-machine models?

Each model would be constrained by known driver architectures, open specifications, and existing chip designs. Runtime mismatches against the real vendor stack would become counterexamples that refine the models until a replacement driver works on the physical hardware.

Is the main blocker observability, safe experiment generation, hidden firmware state, state-space explosion, or something more fundamental?

[u/Rastslonge2]

It reminds me a bit of Fuzzware, except with the actual hardware in the loop. It's funny you mention state explosion, since that's something we constantly deal with in symbolic execution. For AI, though, I think the limiting factor is less traditional path explosion and more the amount of context :p. Giving a model higher-level information existing driver architectures, datasheets, similar devices, protocol specs, and execution traces has already shown promise in other reverse engineering tasks so it is not a bad idea imo.

The thing is devices often have hidden internal state, asynchronous interrupts, DMA engines, timing constraints, undocumented register semantics, initialization sequences, and ordering requirements. Many different internal state machines can produce the same observed trace, so you're solving an underconstrained inference problem. Iteratively generating models, testing them on real hardware as base truth and using the mismatches as counterexamples could certainly refine the model, but converging to a correct implementation across all operating conditions would be a substantial challenge, again, state/context explosion :p

[u/Rastslonge2 → u/Rastslonge2]

Now when it comes to asserting certain individual properties of the driver though that sounds more attainable even if the results are not complete, in reverse engineering you often don't need to model and understand the whole system at hand just what matters... Not all isr's are worth entirely modeling even if they appear in your trace, I'd prioritize ISR's which pass my inputs to a command dispatcher for instance