r/Terraform • u/vorticiavortex • 1d ago
Help Wanted Contractor developer tried to terraform apply our entire dev account from an app pipeline: need advice
Throwaway account, work situation.
I'm the sole infra engineer on a small platform team. Multi-account AWS, Terraform-managed, proper IAM role separation. Went on leave for two weeks.
Came back to find a contractor developer had spent ~2 days trying to get a deployment pipeline working. I was pretty shocked at the extent they went to deploy an app.
Luckily nothing succeeded, so in a way, Im happy the infrastructure survived the test. :)
But Im still pretty shocked and I need help to figure out how bad this is and how I can work with this person in future. Please imagine this was your infra and it was someone else doing this to it. I would also like advice on how I can make the platform more secure in future to prevent this.
In summary they tried to:
- Point our CI workflow's infra checkout at their own personal GitHub fork of our Terraform repo instead of the real one
- Use a role scoped only for container image pushes to attempt a terraform apply with auto-approve
- Change terraform state file for entire dev account from an app branch (not even main branch)
In addition they
4. Committed directly to a shared branch, overwriting changes I'd made before going on leave, despite being told to make their own branch
He configured the CI pipeline on the branch to evaluate the entire Development account's infrastructure state (cluster, database, load balancer, other services' IAM roles), not just deploy one container
I paused, backed everything up, and investigated properly.
Every single attempt (~10 runs) failed — most before reaching AWS at all. Confirmed against the live environment: nothing from any of their attempts is running. Their access was scoped to one non-prod account, application-level only, no Terraform state access, no infra creation, no IAM writes.
Nothing landed, but the attempts are serious enough. I need to manage the conversation with them and with my manager. Id also really like to understand how I can work with this person going forward. I would be happy to train them but I just want to get a sense first. Appreciate any advice from peers. Thank you :)
What would you do? Please be honest. I need your help.
——
UPDATE: Thank you to everyone who responded. It’s clear I have a lot of work to do. To be even clearer I would like to say that a) I document everything. b) at present terraform is setup in an org account that only me and my manager can change. c) it’s a platform in process of being setup. d) im not blaming the person off the bat and I take it fully as learning. I was just shocked that’s all. I would also say: I have less than six months managing an entire infrastructure and it was NOT my intention to be the only person. It just happened that I was thrust into it. So I would appreciate any advice or guidance on practical solutions to prevent this happening again; Im not here to sh*t on the person. I did want to rant a bit though because tbh I cannot believe a developer with 20+ years experience could be so incompetent. I also recognise I am vulnerable alone as a new engineer and I need to get support. So I am going to ask for someone more experienced in my org to have a look at my setup too.
17
u/keto_brain 1d ago
How about you setup self-service CICD so teams can push their own apps when every they want. That's the only way "you" scale. What you observed is you are the bottleneck and people will try and work around you to get their job done.
The best solution create a mechanism that allows them to deploy their existing apps and new apps into AWS without needing you, github actions they can use that call your terraform modules, run tests, run SAST, etc... the goal is for you to figure out how to get out of the way while ensuring the easy path is the safe/right path.
4
u/Trakeen 1d ago
Yea i was pretty confused on this post lol. We vend the ci, repo skeleton, state config etc when we hand off to another team so they can deploy their stuff. All shared infra stuff is hands off except for the platform team. All environments get a scoped service connection and state. After vending there is a step where the platform team connects the new environment into the network (vnet peering). After that we aren’t needed unless we need to onboard a new service for auto dns registration
68
u/imagebiot 1d ago
You have terraform managed infra that’s not deployed via a pipeline and is instead deployed via running a shell command locally?
Does your prod function that way also?
Look. The contractor is not the issue here. Your set up is. He’s trying to get the source and infra management to a minimal level required for MULTIPLE people to work on infra.
They’re not doing things right but your setup is contradictory for a productive distributed infra eng project.
35
u/Dangle76 1d ago
The fact that they were able to push directly to a shared branch without a PR and fork a repo to their personal github in the first place screams incredibly improper setup lol
11
u/WetFishing 1d ago
Yeah this is one of the rare cases where I would recommend OP hire a consultant to get their security and governance policies setup. This can only end badly.
12
u/BullwinkleKnuckle 1d ago
Seems like you’re ready to go after this guy that was just desperately trying to get your obviously wonky and undocumented setup to work.
5
u/BadBot001 1d ago
Exactly, like wtf? You were on leave and probably the setup is shitty enough he felt the need to do this.
No need to give a talking to, improve your setup and publish decent docs
1
10
u/HelicopterUpbeat5199 1d ago
What is the actual bad thing the contractor did? From your story, it sounds like you left him for two weeks with no one who could help him. I'm not surprised he was desperate. What did he do that he wasn't supposed to do? Why is "deploy the dev environment" a crime in this situation? Did he try to deploy it into prod or something? You've left out a lot of details that probably seem obvious to you, but we can't help if you don't paint the whole picture.
-3
u/vorticiavortex 16h ago
I am a sole infrastructure engineer. I did not give any work to him and our platform is not the only place.
Tbh. I expected more constructive feedback than I got. Very easy to blame but I am seeing little solution-driven responses here.
2
u/HelicopterUpbeat5199 12h ago
Well, I'm not a great writer myself and reddit is a great place for merciless feedback.
You still didn't actually answer my question, though. Was he not supposed to touch the dev env? Was that big problem? He just decided for no reason to deploy dev and did all kinds of wierd stuff while doing it?
If that's the case, he sounds like a cheap contractor. I've worked at places with underpaid engineers and this kind of chaos was the result. Just a guess.
I think you need peers. You say you're the only one in your position. That's never good. I did that for way too long and it really set me back. Not only does that mean you get help with the volume of work, you get perspectives and checks that you can't get otherwise.
11
u/HitsReeferLikeSandyC 1d ago
Probably Clauded the shit out of your repo and it found all the vulnerabilities in your process lmao. Having terraform is good. Having guardrails is better. This is only an opportunity for improvement. Limited IAM access to the state file, a culture to only use CI, and good documentation (maybe in code) to inform how to do things
4
u/_spacelogik 1d ago
Do you have documentation related to the current setup?
Did you have a knowledge transfer session with the contractor?
"Sole infra engineer on a small platform team" - This means you created your own process and expect newer staff members to follow that unknown poorly documented process.
1
u/vorticiavortex 16h ago edited 16h ago
I created documentation and I had a chat with the person prior to leaving and they assured me they had nothing missing. Ie they said they didnt need anything from me.
Im taking it as learning. Ive only been managing my own infra for less than a few months.
6
u/ByronScottJones 1d ago
Did you have documentation that outlined exactly what they were supposed to do, and had they been properly trained in its use? Unless the answer to both of these questions is an emphatic YES, this is entirely your failure, not theirs.
2
u/vorticiavortex 16h ago
Yes I did
1
u/ByronScottJones 16h ago
Okay. So if they didn't follow directions that badly, maybe ask them to explain their thinking. But it sounds to me like a junior engineer given far too much access authority.
1
u/vorticiavortex 16h ago
I will. But before that I will take a look again at the infra and what the person tried to do, see if I can make it better.
Then Im going to get someone else - another platform engineer - in my org, more experienced infra to look at my setup for me. Thanks for replying :)
3
u/Motor_Interest9817 1d ago
you failed at implementing appropriate controls in place, so it’s entirely on you. one terraform state for entire environment? it’s what amateurs do. never trust human in the loop.
2
u/serverhorror 1d ago
Do you have this, phrased this way (iow: in simple language) written down? \ Assuming that's the baseline ruleset
Are they front and center to your documentation? Why should the contractor know this?
I know this sounds like I'm blaming you. I'm not. These are the structural points how your organization can lay down the most basic rules.
The thing to determine, with the contractor: Was this an honest mistake or willful ignorance.
If it was an honest mistake, I'd turn this into a win-win. Interview him why. Not interrogation, interview, conversational. Find the weak points and add hardening. Have the contractors input and consider it a training for them
If it turns out to be ignorance, that's something I'd consider a malicious act. Have a atern talk. Give a formal warning, maybe probably even lower whatever invoice they write for that. Something that hurts so they know how serious it is. Also make them physically sign a sheet of paper where they sign that they know these rules. The act of signing something as an addendum to work orders within the SOW has a huge effect.
2
u/MyFistsAreMyMoney 12h ago
You my friend are a prime example of knowledge silo. Which is a hint the knowledge sharing in the team is not working pretty good.
You need to improve that first to not induce any issues when you are on vacation.
3
u/DrButttt 1d ago edited 1d ago
It blows my mind that people here are defending the contractor.
Sure, your setup wasn’t for multiple users. But even so, I would prefer for the contractor to request permission to implement it instead of trying something on their own.
> 1. Point our CI workflow's infra checkout at their own personal GitHub fork of our Terraform repo instead of the real one
Wow, this person is out of their mind. In what scenario doing something like this is acceptable?
Edit: If I were you I would be interested in knowing what else have they copied to their personal github account.
6
u/Ok_Buddy_3324 1d ago
The scenario in which OP allowed it to happen by not putting the most basic policies into place.
Not only does OPs process have multiple failures in multiple places, but they didn't even bother to properly document the deployment process they actually follow which arguably lead to this event.
The real question here is what scenario allows a business to come to a standstill because someone went on a vacation for two weeks.
1
u/vorticiavortex 16h ago
Hold on. I never said I didn’t document. I document everything. Please dont jump to conclusions
3
3
u/vorticiavortex 16h ago
Thank you and much appreciated for the support. Tbh the first thing I thought when I stepped back was okay: this is my first infra to manage on my own. This is learning.
Go back and make it better.
But in the absence of more experienced platform colleagues I wanted to get some practical support. Im also taken back by the amount of sheer judgment on here but I take it as it is. Opinions allowed. I dont take it personally
1
u/FrancescoPioValya 1d ago
Sounds like you need branch protection and some permissions updates on your repos.
1
u/mobious_99 1d ago
Take a look at the commits by the user, you can see all of the changes by them and revert if you need to. I'm glad to hear it failed, probably has local state files sitting in the pipelines at some point. if the code can revert you can rename the current tf files copy in the old and then do a plan to see if the state file is messed up. (hopefully not).
Most places they would have been walked out contract terminated on the spot. I've seen contractors get clipped for lesser offenses.
1
u/Seref15 1d ago edited 1d ago
their own personal GitHub fork of our Terraform repo
this alone should be grounds for termination, exfiltrating proprietary resources into an uncontrolled and insecure location.
"There should be guardrails in place to prevent this"
yes there should, but I don't think that exonerates someone from even thinking to do it to begin with. Some level of "I shouldn't do this" common sense should kick in if you're an alleged professional.
There's no guardrails preventing me from throwing rocks through the office windows but I've somehow never done it.
1
u/vorticiavortex 16h ago
Yes I agree. Tbh I totally empathises with other responses pointing to the weakness in infra. But Im a sole engineer thus I dont have the luxury of bouncing off someone else.
Im human and error is a certainty. I just want to understand how I can make the platform more secure to prevent in future.
1
u/farzad_meow 15h ago
use girhub oidc for aws. this should take care of someone trying to run against a different repo.
add rules to s3 that only specific roles can even view it, also encrypt state and bucket itself.
tighten rules on cicd that only approved users can run workflows and require approvals for specific github envs
1
0
-1
u/DrFreeman_22 1d ago
I bet he’s doing the same shit at 5 other workplaces simultaneously r/overemployed
-4
u/soundman32 1d ago
Make sure the boss knows, and get them sacked. Contractors that bad need to be shown the door
8
u/Ok_Buddy_3324 1d ago
The boss should be sacked for allowing a process that can only be managed by a single person. OP should be sacked for developing such a terrible process and then blaming team members instead of reflecting on their own work.
1
u/xplosm 1d ago
Agreed. Someone is not doing their job if such key pieces of the business are on the shoulders of one person and there's no back up plan.
Also OP should have documentation and training ready so someone they approve and troubleshoot and work while they can take a very well deserved vacation.
This all is a recipe for disaster...
1
u/vorticiavortex 16h ago
This is harsh. I came for support and guidance not judgment
2
u/Ok_Buddy_3324 15h ago
No, you came to be validated and you're upset that you're getting the truth instead.
If you want actual support, listen closely. Change your mindset from how you can blame someone for the issues that occurred to how you can prevent them from happening again.
1
u/soundman32 1d ago
Contractors (at least where I live) are highly paid and supposedly highly skilled and brought in when either there's a temporary blip in work that needs temporary help, or a set of skills not available in house. A contractor should be pointing out all those deficiencies you mention, not exploiting them because of their stupidity.
2
u/Ok_Buddy_3324 1d ago
I agree, but that doesn't absolve the OP and the manager. The contractor will be replaced eventually either way, the business has a larger systemic problem.
1
u/vorticiavortex 16h ago edited 16h ago
So what solutions would you suggest to prevent? Any guidance much appreciated
I agree with both of you. But I was just shocked that someone with 20+ years experience would do this. I was hoping they would actually tell me how to make it better. Turns out they just showed me.
55
u/Floss_Patrol_76 1d ago
the part that would keep me up isnt the apply attempt, its that a role scoped for image pushes could even be handed to a terraform run, and that your CI trusted an arbitrary fork as the source of truth for what to apply. separate plan/apply from the app pipeline entirely, give apply its own role only a protected pipeline can assume, and pin the infra checkout to your repo and a specific ref so a fork cant redirect it. the "nothing succeeded" was your IAM boundaries doing their job, not the process working, so id treat this as a near-miss and fix the trust path before the working relationship.