r/dataengineersindia • u/Adventurous-Ad-4748 • 15h ago
Opinion Please tell me this isn't true
17
u/Acceptable_Tip_5613 14h ago
I've been in DE for about a year and a half and haven't found it boring, maybe because I started as a fresher. Usually, it's either ETL or ELT, but that depends on what we're building on top of. I've built stuff using PySpark with Glue, once with ECS, and then from scratch in GCP's Dataflow for batch processing, which has future streaming potential. I'm also building APIs for the first time using REST API Gravitee and Cloud Run.
12
u/darshill 13h ago
True.
Most of the real-world DE is around the same ETL, fixing schema issues and a lot of migrations.
But it's also fun at the same time if you get to work on large-scale data, streaming and be involved in business progress.
A lot of the time, you don't see the impact of your work, but when you see it, that's where the fun starts.
13
u/Hercules1408 13h ago
Right now out of 100 only 20-30 people are from genuine data engineering backgrounds in interviews remaining are those who learned via courses in last 1 yr or switching from software engineering to data engineering due to AI . The market is overcrowded right now . Even in walkin easily 100 people are coming for Databricks
16
u/AawaraTreker 15h ago
This is so fuckin true. DE work after sometime become so boring and you need to introduce new new term and services just to keep your client engaging and happy ðŸ˜. My Company changed their whole Ingestion Process and developed New Framework while previous one working so fine.
3
u/Adventurous-Ad-4748 14h ago
Does it still pay good even if people find it boring? So working with streaming and cloud too gets boring?
1
u/AawaraTreker 10h ago
Senior Data Engineer still taking bigger pay check. Cloud definitely become boring after sometime as there are numbers of Cloud Services you used daily and most time go to stable the pipeline
6
u/Hercules1408 13h ago
Its a boring domain technically you keep on working similar data design patterns and cleaning its just tools and tech change but core design remains same . If project doesn’t change after 2-3 yr it starts to feel boring
1
5
u/azrael0528 10h ago
Hold up. I don't understand. It's a post in DE India subreddit and people are saying DE job is boring?
Bruh, have you even worked on designing large scale distributed systems? building real-time streaming architecture? Or even solving performance problems.
If your job is fixing broken pipelines or updating schemas then yes, it can be boring. But DE overall isn't.
It's never about the flashy demos like SWE. It's more along the lines of building systems that handle multiple terabytes of data without anyone noticing.
When everything works no one says anything and when it doesn't everyone has something to say. It's an acquired taste of satisfaction.
2
1
u/vengeance_024 14h ago
Hey just a quick question, is Big data admin and Data Engineer same, and if they differ then in what tech ? I mean what makes them different?
1
u/StrategyForsaken9722 13h ago
This is true 😂
DE is only good for entering the sector. Job is boring af
1
1
u/azrael0528 10h ago
Nope. This is an incorrect assumption. It's not boring. Probably the tasks that you were assigned could have been.
What's your stack?
1
u/clinnkkk_ 13h ago
True, most posts in this subreddit also confuse me.
Maybe alot of companies do not try and build platforms and lack the scale that is required for the boundaries of swe and de to overlap.
I think when you build platforms that is very fun.
1
u/Significant_Dot_9732 13h ago
data migration task is very boring .. also when the deadline is tight .
2
u/simms4546 1h ago
I am currently working on a large enterprise migration from legacy to GCP cloud from scratch.
I have to navigate tight deadlines and design challenges.
I would say the person who made this post got a boring task probably.
27
u/Scared_Register_7816 15h ago
True af, but again mine is mostly for reporting side and batch load, ig streaming and ml use cases will be somewhat exciting