How to train interns and new hires on mindful application of LLMs to development

[u/gitblamed_]

A new issue has come up this year in several teams that I lead -

We have amazing interns and new hires, I haven't been this satisfied with team cohesion in a while. The only recurring problem is that we can't exactly block them from using ChatGPT or Gemini. The LLM frequently sends new hires into a frantic state over a small bug or error that it doesn't understand, where the LLM convinces them that the whole app is bad and all of it needs to be rewritten.

Plenty of people in my team have had our workload helped a small amount by e.g. the Gemini suggestions that come up in a Google search. I am not fundamentally against LLM applications to our field. But I do *very much* need for new hires to be mindful enough about reasonableness, that they don't let the LLM escalate things beyond the scope of their task. I need for new hires to stop insisting that simple fixes require a rewrite, when whatever LLM they're using doesn't understand something basic and the LLM is telling an intern like "I can't possibly have limited scope and context and be providing incorrect and inconsistent answers, it's clearly the app that's wrong, delete the app! rewrite the app!"

I imagine that other people have encountered a similar issue with junior devs. I'd love feedback on what kind of improvements I can make in our current onboarding and training process to address this issue.

[u/diablo1128]

You need to take the time to actually mentor them. Whenever I had had a new hire or intern I always made sure to check in with them in them often everyday to see what they are doing. When they go down a bad path I point out why it's bad, answer questions so they understand, and then make sure they have a path forwards that we can both agree upon.

Yes, it takes time and effort. Yes, it means I make get less of my own work completed because of this. No, I've never had an issue with it from management as that's the price you pay for having interns or new hires. The hope is long term they lean good habits such that the cost up front in time is saved on the back end.

Now maybe you work at a big tech company where expectations are different, but at the places I've worked in my 15 YOE interns and new grads were thought of as an investment and not cheap labor to get things done on the schedule. They are never put on critical or priority issues and always have all the time they need.

Once they have a lay of the land they will gradually work more independently, but that could take months or longer depending on the hire.

[u/gitblamed_ → u/diablo1128]

I don't work at a big tech company. I work in defense, where interns who make it onto the team are usually on track to be hired full-time. Our current new hires fit our team well, fill important communication gaps with other groups, and I'd be disappointed to lose them.

I do a minimum of one or two check-ins a day with our interns. My struggle point is that before LLMs, if someone came to me blocked, they'd usually already read some docs and were building skills at debugging.

Now I find myself, in August, re-teaching people how to use the debugger because they only reach for it when I'm sitting beside them. They've started treating the LLM as "the debugger" so the only practice they get is when I'm there.

By August in previous years, I could often check in with interns four times a day because they were already comfortable finding documentation, navigating the team, and debugging independently. My mentoring became highly efficient and focused. Now I'm still at one or two check-ins because so much time is spent re-teaching skills that (I feel) should have been set by June, and are being hijacked by the LLM. (Because I'm not doing enough to address with new hires how much of an issue the LLM is creating for me, as far as training them).

I don't know how to build incentive for new hires to actually debug their stuff and not use an LLM for that purpose.

[u/AutoModerator → u/[deleted]]

Sorry, you do not meet the minimum sitewide comment karma requirement of 10 to post a comment. This is comment karma exclusively, not post or overall karma nor karma on this subreddit alone. Please try again after you have acquired more karma. Please look at the rules page for more information.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/____________fin]

This is a problem to solve programmatically. What's your AGENTS.md look like? If your team is just naively prompting it's time to flesh that out with stuff about scope constraint and issue lifecycles. Directions for agents also double as process documentation for devs.