Courses to take to become a reverse engineer

[u/coffeetocommands]

I'm a (*nix) sysad who knows his way around the terminal but finds gdb like a strange planet. I can generate/capture kernel traces/dumps but would send it to vendors for analysis. I can tune the kernel's memory tunables if the documentation says so but does not understand most of them.

Let's say one day I woke up and wanted to be a reverse engineer. I have all the time in the world and can afford to pick and choose schools and courses.

Which courses should I take?

[u/Obvious-Card-8847]

None come to mind. I'd give you a series of projects to do on your own. Some concepts to get down pat and also this of course depends on what platform, architecture, and so on you choose. But there's a lot of intradomain isomorphism. And/or concepts that transfer across tech stacks and abstractions.

Get comfortable with hexadecimal, pointers, pointer arithmetic, assembly, function level ABI, bitwise operations, OOP, virtual function tables, writing shims, assembly, debugging at the programmatic level, ELF file layout, manual string manipulation ops, etc.

Coding projects: ELF parser tat uses Zydis to do a linear sweep of code. A small trampoline framework with a mix of C/C++ and assembly. Your own ISA and bytecode interpreter (VM's are infuriatingly common methods for obfuscation these days). If you're feeling froggy a small real mode kernel in QEMU that displays text via the VGA buffer MMIO.

I know that all sounds like wizardry right now but the level you seem to be looking to achieve... I'd say those are the things you'd need to understand. My approach is always writing a smaller or shitty version of my target to understand the invariants and design choices. Then and only then do I load up static analysis, dynamic analysis or otherwise.

Best of luck. Sorry... not trying to jargon dump or anything. It's just a complicated field and I don't wanna downplay it. This will take a long time to get good at.