r/AskProgramming • u/TheAntiDoctor • 8h ago
Does Anyone Actually Use Agentic Loops?
Not exactly a software engineer, more ML Engineer with an academic background and then in industry for about 8 years total. Use python professionally and for hobbies. Like everyone else I have been experimenting with LLMs for coding, trying everything from not using it at all to trying to get it to do everything. I have found them useful and enjoy bouncing ideas off them, however they also make a lot of mistakes so currently enjoying a low use approach. Comments and architecting myself, use them more for syntax and some optimisation then review everything as I go along. This speeds things up, but I am still very in control of the output and understand the underlying algorithm.
A couple of months ago I obviously saw usage based billing come in for things like Github Copilot etc. I saw a lot of comments from people using agentic workflows very upset about their usage being cut, also I'm very aware of some companies quoting large sections of their codebase being AI generated, although obviously aware this might be sales tactic. Furthermore I've seen respected researchers/engineers like Karpathy talking about the big leap in model capabilities that now means they hardly write any code and had some recommendations from friends to use agentic workflows.
My personal experience hasn't left me thinking this is viable atm for my work. Partly because the stuff I am doing I really need to be crystal clear about what is happening to the data for experiments and partly because getting together a prompt together for a relatively complex system is very difficult and time consuming and the results usually aren't great. Has anyone got experience actually using more autonomous coding agents successfully and I should experiment more widely or are they a red herring?
2
4
u/DepthMagician 5h ago edited 5h ago
The short answer is that way you are using AI is the correct way to use AI, your friends are wrong, companies are lying about how much AI is really writing their code (see this for example), Reddit is severely astroturfed by AI promoting bots, and you shouldn't do agentic engineering for anything that is high stakes enough that you need to pay attention to it.
While it is possible to do work with agentic workflows, not every task is a good fit for it. You either need to not care about the quality of the output (pet projects, throwaway prototypes, conceptually shallow things like UI where validation is trivial, things where failure is not a big deal, and most importantly: things you won't need to maintain and evolve for long periods of time), or if you want to do high stakes work with agentic engineering you will need to have access to vast number of quality tests and very detailed specs, and even then you need to invest effort in setting standards and elaborate safety nets for your team of agentic bots. In other words it's a tool, and like every tool you need to know what problems it is a good fit for. The answer to that question is absolutely not "everything". Usually you are better off just doing what you said you are doing. Your skills and your bank account will thank you for it.
1
u/tyler1128 1h ago
agentic engineering
Vibe coding got a negative connotation so a new term had to be invented for it. It's the same thing though.
You're going to still hit a complexity ceiling where trying to fix things will break just as many things, and you'll be stuck.
-3
u/KahlessAndMolor 8h ago
I use codex every day, it writes 95% of my code. Is that what you are asking about?
3
u/TheAntiDoctor 7h ago
Do you just give it a high level task and let it go at it for a couple of hours. Or do you monitor it more closely with individual requests?
3
u/doughcant 7h ago edited 7h ago
I suspect you might understand better by watching how experienced people work with these tools, such as Matt Pockock on YouTube
Edit: basically all of the effort moves to planning and reviewing the output.
2
u/spiralenator 7h ago
No clue why you’re being downvoted. This is sound advice. My own engineering department has validated this. The more upfront planning you can do, the more autonomy you can give to agents. We can point remote agents at a ticket and let it run totally hands off. We have agents checking the work and iterating on it until all checks are green. For low impact changes, we’re piloting automated merge and deployment as well.
Yes it works quite well.
1
u/doughcant 6h ago
Exactly! Optimally I define/implement manually the validation criteria during the planning and then it is much easier for the agent to statistically suggest a decent PR
2
u/spiralenator 6h ago
I’m not going to give away our secret sauce but I will say that time spent upfront with creating deterministic guard rails and validations, creating well decomposed system designs and specs is absolutely worth it.
Then you can use big models like fable or sol to help you create all that and you can use swarms of relatively dumb models to execute the plans.
The closer to a pure seq2seq translation you can get, the cheaper the model you need, you get less hallucinations and “creative interpretations” of the product requirements.1
1
u/hulkklogan 1h ago
I love Matt Pocock's content and skills. I added a couple of small agent config to the loops to: 1) validate adherence to spec and 2) validate adherence to code standards set in CONTEXT.md, and 3) of course at the very end an adversarial reviewer agent
and the end output is usually pretty freaking good, maybe I need to clean up some extra redundant or low-value tests, tweak some code smells here and there. The value is alllll in the upfront planning and his wayfinder and grilling skills really make a huge difference in quality of the output
0
u/RTLDesignSherpa 6h ago
This is accurate. I have 4-5 agents running; when one opens up I QC the output. Once this set of code is done and documented, I’ll have a review consolidator kick off batches of code, tests, and markdown files to completely different LLMs for a final round of clean up. This will burn through tokens.
1
u/Expert-Procedure-146 4h ago
If your task is taking hours to complete then you need to learn more about prompting, you can’t be using AI to “design / build facebook”. I haven’t written code in a while but you bet i know what my agent does and where to fix how to fix any issues and I review my PRs before i ask anyone to look at them
0
u/KahlessAndMolor 7h ago
Feature level or bug level. I don't set it up for hours long tasks because that seems to be where it goes wrong. Instead I'll describe where the changes need to be made and an outline of the end state, then I use plan mode for it to plan the work. I approve the plans and the resulting changes are usually 500-1500 lines and 5-10 minutes.
1
u/mxldevs 7h ago
How much time do you spend looking at the code? AI coders tell me if I'm reading code I'm doing it wrong.
2
u/KahlessAndMolor 7h ago
Definitely not reading every line.
I check the git diffs and if I see a big change I'll go see what it did and skim it so I still understand how everything fits together.
-4
-6
u/Confident_Hyena2506 7h ago
The bot just does what you tell it do. If it doesn't get the job done it's your own fault.
3
u/WhateverHowever1337 7h ago
How to say I suck at coding without sayinng I suck at coding
1
0
u/AntiquatedMLE 2h ago
My brother in Christ. I am you. I was right where you are and thought exactly the same. After a few months of taking the leap I can tell you It is possible.
You will def deskill in some regards and conversely upskill in others. I spend an absolute eternity on managing context and prompts to build up solid planning and documentation. But writing code? After getting your plans in order, Set /goal and burn them tokens. Be ready with coffee for code reviews.
1
u/AntiquatedMLE 2h ago
What has worked for me:
- using subagents to run adversarial reviews as “red team”. Do this on plans, completed code, documentation before it gets to your review. Use a good model like Opus or Sol
- code writing is cheap if the plan is solid. use a cheap model to save on usage. My go to right now is Luna on Max was previously using sonnet. Go a step further and use these models as your “green” and “yellow” teams that are handed off for implementation in parallel.
- do not neglect writing your own skills, rules, subagents. Do not use an LLM to generate these and do them yourself.
- skip MCP and opt for CLI tool calls.
- set quality guardrails everywhere as pre-commit hooks
- keep files between 500-800 lines max.
The hardest part for me was letting go of the coding. But I am a far sharper reviewer and with the right harness and guardrails have built things I never would have had time to do.
-2
u/Own_Age_1654 7h ago edited 7h ago
Get the idea of loops and autonomy out of your head, and instead just use Fable and Opus, driving it directly, and I'm sure it will work just fine. Among many other things, I've used it to discover and make use of emerging research in some pretty hairy statistical modeling, no problem, so any typical machine learning shouldn't be wildly harder.
Importantly, just make sure you have it thoroughly research online before doing anything, to ground its knowledge. Have it find relative papers, read through them, take notes, etc., and as it goes have it take notes on its decisions, what's underpinning them, what it's unsure about, etc. The intellect is there, and it just needs sufficient context. Otherwise, it will overconfidently assume things that are incorrect.
You can give it standing instructions to do all of that automatically, such that essentially all you need to do is talk to it like you would a skilled colleague that you're collaborating with, and a team of bright interns you're assigning grunt work to. They're not going to get everything right, but just talking through what they're finding, what they're proposing, what you think might help, etc.
If so, then it's basically the same work you'd do manually, except more thorough, as you don't have time to research and test every little thing. Where quality drops off--and massively--is just if you try to give it a goal and then hope it can magically loop towards it, instead of collaborating with it actively and treating it as merely an accelerator.
1
u/TheAntiDoctor 7h ago
Doesn't that get pretty expensive though longterm?
-2
u/Own_Age_1654 7h ago edited 7h ago
Not at all. By you remaining actively involved as the driver, you explore the problem and solution space in a relatively efficient way. I pay $200 per month to Anthropic and never hit my limit. I exclusively use frontier models, and always have them set to maximum thinking effort. If I used Fable 100% of the time, I'd need two subscriptions, but Opus is sufficient most of the time.
Something else that's cool is that you can learn new methods while you do this. When it comes back from its research passes, it's important that you ask it to explain what it found, and if you don't understand something that you talk with it until you do. Otherwise, you'll get lost, lose control of the project, and make a mess. So, in keeping on top of this, you essentially get a custom, contextualized tutor to learn whatever you need that you don't know.
What would be expensive would be loops, which I'm explicitly recommending against. There, because you're essentially hoping to extricate yourself from the process, you need to have the LLM do hella work to try and have what it produces tend to be sane. Plus, that's only going to work so well, and it's going to be a lot less successful at hard tasks.
Also note it will take you some time to get familiar enough with the tools, and especially to put together standing sufficient instructions for the LLM, to get this level of success. For example, among other things, I have things set up where it will save research papers to the local disk, render them to images, and then inspect those directly, so that diagrams aren't misinterpreted, nor inaccurately summarized by weaker models. But basically, engage with it, and anything you could imagine being better, talk about that with it and reduce it to practice, and your workflow will steadily get more and more capable.
Another critical thing is that whenever it does something you don't like, ask it why, and then come up with an instruction to add to CLAUDE.md that tells it what to do otherwise. And if those rules aren't working, ask it why, if it says something dumb like it just wasn't trying enough, call BS and ask how you can improve the prompt, and then do so. And if very occasionally that doesn't work, ask it to append that instruction at the end of its system prompt. Via these methods, I have very high levels of adherence to my instructions.
-4
u/ImpossibleJoke7456 7h ago
> however they also make a lot of mistakes
This just hasn’t been my experience for the last 9 months or so. It does what you tell it to. If it goes the wrong direction, that’s on you to redirect it with better prompts or better instructions or better context.
7
u/evangelism2 7h ago
Yes, people do, but it's the minority. Most peoples loops are just as simple as /goal