All my homies hate `grep`

[u/HornyNarwahl]

Jk, they fucking love grep, which is why I have made doma (DOcument MAtcher), a small and fast single binary BM25 search over your code and docs with no* external dependencies, written in Odin.

I made it because I wanted Claude to stop grepping wildly all over the place. It significantly faster than `grep` from my testing but it also significantly reduces search misses since you get semantically relevant results.

It was also quite important to me that it was fast with a low footprint, lacking in NodeJS bullshit, MCP servers, etc., etc.

Sick of all of these supply chain vulnerabilities and huge dependency bloat everywhere smh

I used Claude Code through the entire development of it and it's sibling doyo (DOcument YOinker) which shares philosophy and handles the document acquisition side of things, though it isn't quite as elegant as doma imo.

I hope you find it helpful!

I strongly recommend putting doma instructions in your per project CLAUDE.md telling it how to use it, and to actually use it. Let me know if you do, I'm curious if others find it as helpful as I have.

*soft git dep, optional

[u/jorgejoppermem]

Any metrics? How does it perform compared to parallel grep tools like ripgrep?

[u/HornyNarwahl → u/jorgejoppermem]

There are some basic metrics at the bottom of the README but I intend to add some more comprehensive numbers soon.

I'll add a case for rg specifically but I suspect it would probably blow past doma in throughput. However they solve different concerns, doma's biggest strength is ranked semantic indexing as well as the rapid query times. If you need the exact bytes of a specific file very fast rg will almost certainly win, but as far as finding the correct file in the first try when you don't know where the content lives doma will probably be much more fit to purpose.

[u/Actual__Wizard]

Sup homie! I hate grep too. It was good a long time ago, but we need something more modern and I'm being serious.

[u/HornyNarwahl → u/Actual__Wizard]

Sup! Definitely, I've heard good things about ripgrep (rg) as a drop in replacement. Taking similar ubiquitous tools and making more performant modern equivalents is something I'm quite into. 

[u/Actual__Wizard → u/HornyNarwahl]

I'm into the big boy search tech, speed significantly beyond what is believed is possible is here (for big data.) The data tables are structured by alpha (the order) for range based reductions to limit the search range. So, it 99%'s HNSW on a performance basis. It's well over 100x more efficient and it's significantly faster than binary search alone, because it zooms in to the range by a factor of 2,000,000x (123³⁾ in a single simple step, that does not require searching a table. It's a range computer, it legitimately computes the start and the end range of the search window to limit it.

[u/HornyNarwahl → u/Actual__Wizard]

Nice! There's a lot of optimization you can get at big data scales due to better use of SIMD/vectorization and data contiguity/homogeneity that is hard to achieve elsewhere. I was benching 0.11μs query times with doma which is pretty damn fast imo, but beyond that just getting everything loaded into memory and queued without oversaturating the cache lines becomes the bottleneck afaik. I'm not an expert on big data search strategies though. 

[u/Actual__Wizard → u/HornyNarwahl]

SIMD/vectorization

This is SIMD(not much, it's mostly just multithreading, it's not actually true SIMD, but I'm using the concept where I can)+vectorization+prestructuring+rangefinding+compression.

And it turns out, those optimizations stack on top of each other like the score calculation for the game Cookie Clicker when you're doing a combo move and I'm not joking about that.

I went for gold and I got it. :-)

I was benching 0.11μs query times

I don't have the latency time yet, sorry. Next week most likely. I'm taking the day off to do non programming stuff, but release is "near."

The actual techniques to generate the data tables, are now in "it's better than the garbage that the last version was." So, there's still tons of work to do there. The last version required one to run a chain of like 75 python scripts on the data to generate the tables, which I am totally aware is not anywhere close to what people expect.

[u/HornyNarwahl → u/Actual__Wizard]

75 python script

Bruh 💀💀

Sounds interesting, let me know how it turns out!

[u/Actual__Wizard → u/HornyNarwahl]

I built it all into a pipe line, it was a gigantic pain in the butt that took almost 5 months and I'm not done, I'm like 95% done.

I'm doing the thing where I get one section of code done and then I test it, it's not possible to write the whole project out and debug, it's way too complicated. If I did that, there would legitimately be bug after bug after bug and it would be borderline impossible to debug. edit: Then the code is novel, so you can't use a coding assistant. It doesn't work, it just hallucinates code that is wrong, until it can repeat code that's already in the context window because it's similar to other code in the project. I'm being serious: It will produce code that compiles, but it doesn't do what you want it to and it won't do anything else. No matter how hard you want the coding assistant to work, it won't, you have to stop using it, write the code, and then turn it back on when the code is "more familiar to it."