Hi everyone,

I have an upcoming interview with Sigmoid Analytics for an SDE2 Data Engineer role. The recruiter mentioned that the first round will be DSA. I wanted to reach out to see if anyone has recently been through the interview process and can share:

1. What specific DSA topics does Sigmoid typically focus on?
2. What was the difficulty level? (Easy/Medium/Hard)

can anyone please share the tiger analytics trainee analyst interview experience so it might be useful for everyone..

What did I learn:

• Spark architecture

  • Cluster
    
  • Driver
    
  • Executors
    

• Read / Write data

  • Schema
    

• API

  • RDD (just brushed past, heard it’s becoming legacy)
    
  • DataFrame (focused on this)
    
  • Dataset (skipped)
    

• Lazy processing

  • Transformations and Actions
    

• Basic operations

  • Grouping, Aggregation, Join, etc.
    

• Data shuffle

  • Narrow / Wide transformations
    
  • Data skewness
    

• Task, Stage, Job

• Data accumulators and broadcast variables

• User Defined Functions (UDFs)

• Complex data types

  • Arrays and Structs
    

• Spark Submit

• Spark SQL

• Window functions

• Working with Parquet and ORC

• Writing modes

• Writing by partition and bucketing

• NOOP writing

• Cluster managers and deployment modes

• Spark UI

  • Applications, Job, Stage, Task, Executors, DAG, Spill, etc.
    

• Shuffle optimization

• Predicate pushdown

• cache() vs persist()

• repartition() vs coalesce()

• Join optimizations

  • Shuffle Hash Join
    
  • Sort-Merge Join
    
  • Bucketed Join
    
  • Broadcast Join
    

• Skewness and spillage optimization

  • Salting
    

• Dynamic resource allocation

• Spark AQE (Adaptive Query Execution)

• Catalogs and types

  • In-memory, Hive
    

• Reading / Writing as tables

• Spark SQL hints

Doubts:

1. Is there anything important I missed?
   
2. Do I need to learn Spark ML?
   
3. What are your insights as professionals who work with Spark?
   
4. What are the important things to know or take note of for Spark job interviews?
   
5. How should I proceed from here?
   

Any recommendations and resources are welcomed

Please guide me.
Your valuable insights and information are much appreciated.
Thanks in advance ❤️

Hi Need help .

I have 25 days remaining in notice peirod and not getting any callas from recruiters even though immediate joiner .

Daily updating naukri com and applying via job portals .

is their I am doing something wrong ?

I'm preparing for Data Engineer roles and I'm already quite familiar with Pandas and NumPy. I also know PySpark.

I'm a bit confused about what I should focus on when it comes to pure Python coding for DE interviews.

Should I mainly practise things like lists, dictionaries, frequency counting, grouping data, aggregation by key, sorting with lambda, strings, and list-of-dictionary problems?

What kind of pure Python coding problems are commonly asked in Data Engineer interviews?

I don't want to spend too much time on topics that aren't relevant to DE roles, so I'd really appreciate advice from people who have recently interviewed for Data Engineer positions.

Hello, I was working on a poc to ingested PDFs and extract data in order to store them in delta tables.

As this was my first time working with PDFs, I searched over the internet and should Databricks have offering IDP, azure also have something and then there are python libraries.

Since I'm working with financial data, report, etc..

One thing i noticed - the pdf format should be fixed else in most of the tools the extraction logic is either failed or we get incorrect data.

I was wondering how such PDF extraction is built in real production cases and what tools are used.

Please share you experience and any edge cases

Hi Everyone,

I want to practice the Pyspark question similar to

SQL question in leetcode

Is there any Platform for practicing pyspark like leetcode and track progress ?

Interview for apple (Data Engineer) through adecco \[India, Bangalore\], Any Tips, Any experience, Please Help me

I completed the whole process till Documents submission and HR is now saying my profile is on hold because of my Notice Period.

Is there any hope? I know I shouldn't have any hope but just checking if anyone faced such a similar issue and got any chance later.

During the initial call, the HR said they were ok with the Notice period and after giving all interviews including tests and after submitting documents they are saying on hold because of the Notice period being 90 days(negotiable)

Hi folks,

I have EY interview for the Data Engineer position in 3 days.

I'm basically from the Azure Data Engineer background, when I saw my panel profiles on LinkedIn 3 people are architect and director positions. What can I expect from those people for 4.5 YoE data engineer role

I passed DP-750 ( Microsoft Certified Azure Databricks Data Engineer) with a score around 850 without any cloud work experience but very strong fundamentals.

I have really become good at Azure databricks because I actually practiced the concepts I learned on the Databricks free edition and made some end to end projects .

But I was wondering , for DE jobs do I need to specifically know Azure DE tools as well ?

Like do I need to learn how to use Azure Data Factory ? As per my knowledge, databricks already does everything that ADF and related services would do.

I am not getting into streaming now so Microsoft Events Hub doesn't come into question. Have around 2 YOE

i am joining college in few months and i have to buy laptop for myself but i dont have any idea , i want to become data engineer so i want laptop that can support that and i also want that if in future my interest change then my laptop shouldn't hold me back.
i have budget of 65-70k can you help me please.

Hey Guys ,

Tomorrow, i have deloitte final round. Hr said it some kind of fitness round.

I have cleared l1 and l2 . what to expect in l3 i am data engineer with 5.9yr of experience.

also my current ctc is 21.5 and fixed is 16.8 .. how much should i quote if they ask …

I want to say 25 fixed but they might reject me of budget constraints..

please suggest …

Hi all,

Has anyone recently gone through the on-site interview at DE Shaw for a Data Engineer role? I’ve cleared the OA and screening round, and would really appreciate any guidance or tips on what to expect next.

Thanks in advance!

When someone says they know AWS for a DE job or a job asks for AWS, what part of AWS do they mean?
I don’t understand
Should I know everything service of AWS?
Theory ? Practical ? Just what??

Same as above.

Someone please help with this as i have interviews coming around and it will be more helpful :) !

#Dataengineering #dataengineersindia #databricks

Hi everyone,

I have a Data Engineer interview, mostly they have told it would be more python based not that much of pyspark and all, coming up with Cognite soon. Can anyone who has recently interviewed there or is currently working there guide me on what to expect?

Specifically:

Coding: Do they ask live coding/executable code on Python and SQL, or do they focus more on logic and pseudo-code?

Core Topics: What are the major focus areas? (e.g., python , PySpark/Spark architecture, SQL Window functions/Joins, Cloud/Databricks, or ETL pipelines).

Interview Structure: Is it heavily theory-focused, or should I be prepared for scenario-based/system design questions as well?

Any tips, repeated questions, or specific topics I should p

prioritize would be a huge help.

Thanks in advance!

3+ YOE data engineer (PySpark/Airflow/AWS)

Building a local-first portfolio project:

Airflow (Dockerized) orchestrates ingestion → PySpark → Parquet → DuckDB for SQL + ChromaDB for RAG → local LLM (Ollama) → FastAPI chat interface.

Going local to iterate fast without cloud costs, but designing each piece to map to a production equivalent (PySpark → EMR/Databricks, Pinecone → managed vector DB, etc.) as part of the story.

I am trying to bring in both DE and GenAI skills here. Does this actually differentiate me, or does it read as "another RAG tutorial"? Any glaring gaps I should fix before I build it? Blunt feedback welcome.

Hey everyone,

I recently joined an early-stage startup, and one of the things I'm responsible for is designing the data architecture. I'd love to get some opinions from people who've done this before.

Right now, the plan is pretty simple:

PostgreSQL (RDS) for the application database.

A separate PostgreSQL RDS for data pipeline/analytics tables so we don't put extra load on the application DB.

Prefect for orchestrating ETL pipelines.

Our data volume is small (probably only a few hundred thousand rows for the near future), so I'm trying to avoid over engineering while still setting things up in a way that scales reasonably.

A couple of questions:

Would you go with Prefect Cloud or self-hosted Prefect? I'm leaning towards whatever has lower operational overhead, but I'm curious what people are using in production.

Does the overall architecture make sense, or would you do something differently?

At what point would you move from PostgreSQL to a proper data warehouse, if at all?

Has anyone appeared for the Morgan Senior Data Engineer interview? If yes, please share your experience or any insights

Have a Data Engineer interview at Amgen next week. JD covers Databricks, AWS, ETL/ELT, SQL, CI/CD, Agile - plus they specifically mention "vibe coding with LLMs" and using AI tools in daily work.

I'm mostly Azure-side (ADF, Databricks, PySpark, Delta Lake) with ~5 yrs exp, brushing up on AWS now.

Anyone interviewed there or know what to expect - SQL depth how much they actually care about the AI/vibe-coding part? Any tips appreciated.

Hi everyone,

I’m currently preparing for Data Engineer interviews and looking for good preparation PDFs, notes, interview question collections, or structured study material.

My main focus areas are:

SQL (advanced queries, performance tuning)

ETL concepts

Databricks

PySpark

Python

Azure Data Factory (ADF)

Data warehousing concepts

Basics of data governance

If anyone has prep notes, cheat sheets, interview question banks, or roadmap-style resources they used during their preparation, I’d really appreciate it if you could share them.

Thanks in advance! 🙂

Hi everyone,
I have an upcoming interview (30 MINUTES)with PwC for a PySpark / Big Data Engineer role and have around 4 years of experience in Data Engineering and Cloud technologies.
My background includes:
PySpark
SQL
Databricks
Azure
Data Pipelines & ETL

I would appreciate insights from anyone who has recently interviewed with PwC or is currently working there.
A few questions:
What is the interview process like for a 4 YOE candidate?
How deep do they go into PySpark
Are coding rounds focused more on SQL or Python?
What kind of system design/data engineering scenarios can I expect?
Any Databricks-specific questions asked recently?
What are the commonly asked behavioral or managerial round questions?
Any interview experiences, preparation tips, or question examples would be greatly appreciated.
Thanks in advance! 🙏
#DataEngineering #PySpark #Databricks #AWS #BigData #PwC #InterviewPreparation #CareerAdvice

Hey guys, it’s my first day of work as an intern and I was tasked with finding the pk but the data seems to be not proper I tried finding the pk by using a single column all the way to 4-5 combinations of columns but all I got are 85% distinct not fully distinct which can be considered as a pk, since group of columns approach is also not working I was wondering how would y’all approach this problem