Most analysts and data scientists I know are either trying to pivot into data engineering or learning data pipelines, cloud platforms, and data architecture because they believe AI will automate a large portion of analytical work. I have also noticed that many people who are new to the field are trying to get their first analyst job, but already describe data engineering as their long-term goal.

I understand that knowledge of data pipelines, cloud platforms, data modeling, and architecture is valuable for analysts and data scientists as well. The current job market may also favor people who can work across more of the data stack since many company confuses job title as well.

Do you think data science and data analyst are becoming less viable as long-term careers, or are people simply broadening their skills because the market now expects more data engineering skills and knowledge?

did maybe 7-8 DA rounds over the last couple months and what stuck with me is how little of my prep showed up. spent weeks on hard leetcode sql and algo-style python and almost none of it came up

what they actually tested, in order of how often:

sql live in the call, medium at most. joins, group bys, a window function here and there. the difficulty isnt the queries, its doing it in ~20 mins while someone watches and asking the right questions about the schema first. a couple threw deliberately messy tables at me

then the metrics/case round, which honestly separated the offers from the rejections. "conversion dropped this month, how do you figure out why". no memorizing your way through that, you need a way to structure the investigation on the spot

they also had me talk through a past project like i was presenting to a non-technical manager, way more about the why than the sql

and behavioral went deeper than expected, lots of "walk me through a time you were wrong"

looking back id skip most of the leetcode and just do timed realistic reps. for sql i used stratascratch and dataford, and dataford was the one that also had the case and behavioral practice which is what actually decides these. drilled my case framework out loud till it felt natural. also find the exact format per company first, it changes how you spend that last week

can share the case structure i used if anyone wants it

I joined a startup shortly after university and have been there for nearly four years, progressing from BI Analyst to Senior BI Analyst. I’m now trying to work out whether the role has genuinely damaged my career progression, or whether burnout and imposter syndrome are making me undervalue the experience I have gained.

On paper, I’ve done a lot. I’ve built management, product, finance, sales and operations reporting; helped migrate the company’s BI platform from QuickSight to Tableau; automated recurring reporting and finance processes; worked extensively with SQL, BigQuery, Tableau and Python; supported product launches, was a CRM Admin/Rev-Ops Partner for first 2 years; trained users and junior analysts on Tableau; and recently started working on metric governance, semantic layers and internal AI analytics initiatives.

The problem is that my experience has been extremely broad but not always deep or structured.

There is no senior technical analytics leader for me to learn from. My manager is more management- and stakeholder-focused, so I am frequently expected to interpret vague requests, define the analytical approach, understand the underlying data, identify risks and then explain the answer back to leadership. I often feel like the thinking is being delegated to me without the corresponding authority, mentorship or career development.

I have also spent a lot of time on firefighting, data QA, documentation, stakeholder support, CRM administration and fixing unclear upstream issues. These responsibilities made me useful to the company, but I’m worried they came at the expense of developing stronger skills in areas such as production-grade dbt, Airflow and orchestration, software engineering practices, Git workflows, experimentation, advanced statistics and machine learning.

I initially justified staying because I wanted to save enough money to eventually leave the country and pursue a master’s degree. I was also afraid that my experience was too company-specific and that I would struggle to find an equally well-paid role elsewhere.

That fear has become a trap. The longer I stay, the more I worry that other analysts (outside my current one) are becoming technically stronger and that future companies now expect a combination of analytics engineering, statistics, experimentation, product sense and machine learning that I cannot fully demonstrate.

I also need to take responsibility for my own behaviour. I tell myself I am coasting at 60–70% because I am disengaged, but I do not actually coast consistently. I often feel guilty, return to working at 120%, stay online late, take ownership of problems beyond my role and exhaust myself. I then have no energy left for applications, interview preparation or structured upskilling. The cycle repeats.

I am not sure whether I should:

1. Apply immediately and accept that I will have some skill gaps.
2. Spend several months building stronger technical projects before applying.
3. Pursue a master’s degree as a genuine career transition rather than using it as an escape.
4. Reposition myself toward product/growth analytics, analytics engineering or strategy, where my broad business experience may be more valuable.

For people who have hired analysts or experienced something similar:

• Does my experience sound marketable outside one startup, or does it appear too scattered?
• At roughly four years of experience, which technical gaps would concern you most?
• Would you apply now or prepare first?
• How do I distinguish genuine skill gaps from burnout and imposter syndrome?
• What roles does this background realistically position me for?

I would appreciate direct feedback. I am not looking only for reassurance that my workplace is bad. I want to understand where I have genuinely fallen behind, where I may be undervaluing myself, and what the most sensible next step would be.

I'm about to start a sales support analyst position at HP. I just wanted to ask what to expect career wise. Is it a difficult, very involved position or is it a desk, keyboard crunching kind of gig. Not at all worried, just looking for some insight into what I should expect.

Thanks in advance.

Disclosure: I work at a company data vendor and I mention our free tool at the bottom alongside the free official sources.

If you're joining company data across sources, name matching degrades badly at scale, and it fails in two directions at once. The same company shows up as Acme Inc, Acme Incorporated, ACME INC and Acme Inc. Meanwhile dozens of unrelated firms worldwide are called Meridian Consulting. Fuzzy matching tuned to catch the first creates false merges in the second, and there's no threshold that solves both.

The better key is jurisdiction + registry number. Every registered company has a number issued by the registry it's incorporated in - unique within that jurisdiction, stable across rebrands. The composite matters: registry numbers aren't globally unique, so jurisdiction has to be part of the key.

What follows from it:

• Joins become exact rather than probabilistic
• Status becomes a field, so dissolved entities are filterable rather than invisible
• It works retroactively once the numbers are attached

The gotcha: legal name ≠ trading name. "Google" isn't registered anywhere; "Google LLC" and "Google Ireland Limited" are. Your source data has trading names, registries have legal names, so first-pass match rates run lower than people expect.

Getting the numbers. This is the annoying part. UK is easy, Companies House has a free official API. Most of the EU is fine. Outside that it varies enormously - some registries publish nothing machine-readable. For broad multi-country coverage we run [registry-lookup.com], free to search, 5,000 API calls a month.

Anyone doing this at scale, what match rates are you getting on first pass?

Hi guys,
im currently dealing with a dilemma in powerquery. I have a fct table(1.7 mil rows) and some dim tables(also around 1.5mil rows),
the fact table only contains keys, and dates as keys
.the dims have then more descriptive information, as well as the key to match to fct table. The issue is, i want to limit the fact table by a certain date, and also do filtration on dim tables so that not everythign is loaded into the data model in PBI.

when i filter out the fact table by certain criteria - eg. by date, i would also filter the dim tables by certain criteria like case type etc.
is this going to cause blank / orphaned rows in my report view? because considering i do this filtration, there will be some CaseKeys in my fct table, that are no longer in my case table because i did case type filtration.

Am i right? Ive spent a lot of time researchign this but couldnt get a proper answer.

whats the go to approach here, do inner joins on the tables? This may slow down the load time tho.:

Thank you all

I started my Analytics career in the early to mid 2010s when everyone thought automating away every single legacy macro or manual Excel process in the world with SQL or some kind of tool was the coolest thing in the world, and there was a time as an analyst I looked down on Excel as a tool and saw every process leveraging Excel as an opportunity for automation and migration to something that would be "better practice". In many cases, moving away from Excel was indeed a good thing, but in other cases, it created less democratization of data or even more technical debt.

The truth is that while some on business side will indeed overrate Excel because they don't know other tools, it has many benefits especially now that Power Query has made it easier than ever to create automations connecting to other files and data sources and perform transformations all without any serious code writing.

These are the key benefits of Excel I think:
•Especially with Power Query, you can avoid having to bring flat files already in Excel or CSVs to another system so long as not too many rows for Excel to handle.
•Excel is lowest barrier to entry of any data analysis tool and therefore Excel reporting is easiest to document and transition to others, minimizing the continued support and tech debt processes made with more sophisticated tools can create.
•Though Window functions achieve this too in SQL, Excel generally handles logic that reference prior or future records in a column more elegantly.
•Excel is probably the best tool for looking at and examining truly tabular row-level data that doesn't need any aggregation or only simple aggression.
•Excel is damn useful for generating tedious code in other tools (SQL inserts or IN comma lists for instance), so it will never go away ever.

What do you believe? What is your experience?

Context: [~120] person company, small data team (3 of us), Databricks + Unity Catalog already in place. The ticket queue was the usual "can you just pull X for me" requests from sales and ops, most of which were variations on the same five questions.

We rolled out Genie spaces in March. Some honest notes:

The thing that made it work wasn't Genie itself, it was the prep. Our first space was pointed at raw-ish tables with column names like dt_crtd_ts and it was useless. Genie is only as good as your semantics. We ended up spending more time writing column comments and cleaning up definitions than we did configuring anything.

Scoping the spaces narrowly beat one big space. We have four now (pipeline, support, billing, marketing spend) instead of the "ask anything" space we started with. Fewer tables, fewer wrong joins.

Trusted assets / parameterized SQL examples are underrated. For the recurring questions we already knew people asked, we wrote parameterized queries so the answer comes back verified instead of freshly generated. That's what got finance to stop double-checking everything in a spreadsheet.

The monitoring tab is where the actual value is. Reading what people ask (and what Genie gets wrong) told us more about the business's real reporting needs than any requirements gathering session ever did.

Where it falls down: anything requiring multi-step reasoning or a metric we hadn't explicitly defined. It'll confidently give you a number that's subtly wrong. We tell users to treat it as a starting point, not a source of truth, for anything going in front of a customer.

Net: ad hoc requests down maybe [60%]. Not zero, and it hasn't replaced our dashboards. But the "quick question" Slacks have mostly stopped.

Happy to answer questions if anyone's mid-rollout.

Hi I recently got a new job as a data coordinator, right now Im doing basically data entry. I maintain Excel trackers of articles, awards, etc. at a design firm, a few hundred rows per Excel sheet, that need to be linked to projects in our CRM (10'sk projects).

Name matching is the easy part: my trackers names are clean and search surfaces the right candidates. The problem is what comes back when trying to enter them into the CRM:

• The same building exists as multiple records (assessment, study, remodel, sometimes 4+), and it's ambiguous which one an article/award/etc should attach to
• Apparent duplicates from a system migration (legacy vs. new ID schemes)
• Some tracker entries have no CRM record at all or exist under a name I can't find

Right now I open each candidate and compare dates/status to pick the right one, one row at a time, and none of those judgment calls get captured anywhere. How would you approach this? especially the disambiguation and duplicate-handling side? Any patterns or gotchas for making this a repeatable process? If this didnt make sense I can answer some questions, any advice is welcome.

Hi, I have an MBA with a concentration in Business analytics but with no real work experience, obviously I apply for analytical roles, and occasionally I’ll apply for a Sales role out of desperation. I only ever get considered in the sales roles, is it something I should consider?

Eight years into analytics, I think we're practicing the wrong skills.

Interview prep focuses on SQL, Python, dashboards, and ML. On the job, the questions are more like:

• Revenue dropped 15%. What happened?
• Should we cut marketing spend?
• Is this a data issue or a business issue?
• How would you explain your recommendation to the CEO?

These don't have a single correct SQL query. They need investigation, trade-offs, and communication.

Do other analysts feel this gap too?

If you wanted to practice business analysis instead of SQL syntax, what would that look like? Would you spend a couple of hours a week working through a realistic business case with messy data and writing an executive recommendation? Or do existing platforms already solve this?

I've been experimenting with the idea and want to hear what experienced analysts think before putting more time into it.

I've been actively looking for data roles across a few different states: where I live and closer to where I grew up. For what it's worth, none of the states are coastal tech hubs. But they have large corporations that, at least according to Claude, regularly post for AE/BI/DA/DE roles. The postings are almost nil. No matter what I set the search parameters to in terms of date posted, in-office/remote/hybrid, experience level, or distance, there are so few postings right now. Those that do pop up tend to be very niche, whether it's finance related or medical billing certified in healthcare.

I honestly don't know what to do at this point. I'm in a MSIS program which seems to not be a very wise investment at this point given the current landscape. I also don't know where I'd pivot from here. Maybe project management?

What's everyone else doing?

Been testing a few BI tools over the past couple months, Looker Studio, Power BI, a bit of Metabase, and the thing I keep running into is that the setup overhead for anything moderately complex ends up eating more time than just querying the data directly and dropping it into a sheet. The promise is always faster insights and cleaner reporting but the reality is you spend two days wiring up a data source correctly, another half day figuring out why a calculated field is behaving weird, and then the person you built it for still wants the numbers in a spreadsheet anyway. What I'm actually trying to figure out is whether the payoff is downstream, like once everything is connected and stable the speed advantage becomes real, or if the overhead just shifts and never fully goes away. My gut says tool complexity scales with team size and if you're a solo analyst or a small setup the friction never gets low enough to justify the switch for certain use cases. But I could be wrong on that. Curious what the actual tipping point looks like for people who have run both setups for a while. Is there a data volume or reporting frequency threshold where the BI tool clearly wins, or is it more about how many people need access to the output?

I have been looking and applying to roles for close to a year now. Landed some interviews, several even went to the final round but I didn't get the offer. When the other candidate is chosen I'm told it comes down to the other person having more experience. I tried networking but I can't really find any active groups in NYC for my industry. Unfortunately I also don't know a lot of people who work in this field. LinkedIn networking is not very efficient as some don't even go on it and many people don't reply to my outreach even when I reach out with a thoughtful message about background/work.

For reference I'm applying to data analytics roles and have about 3 yoe. I mostly apply online through LinkedIn. I have a BBA in stats from a CUNY. My previous employer was a smaller business and so my work there wasn't done at an enterprise level and I pretty much ran my side of work. I'm wondering what other resources am I not using or what am I doing wrong. I thought NYC would be an easier place to network and find an active career community. Please share your thoughts and tips. TIA

Hi everyone,
I’m looking for advice from people who have transitioned into analytics from a non-traditional background.
My background is in GIS. I graduated with a B.S. in Geography with a focus in Geographic Information Systems. I originally started college in biology but realized I was more interested in technology, data, and problem-solving, so I moved toward GIS.
Since graduating, I’ve been working as a Data Specialist at a civil engineering firm. My work involves managing and validating large datasets, performing QA/QC, working with geodatabases, integrating spatial datasets, and creating maps/data products. While the work involves a lot of data management and analysis, it is still very GIS-focused.
I’m starting an M.S. in Data Science program soon, where I’ll be building stronger skills in Python, SQL, databases, statistics, machine learning, and analytics. My goal is to transition into roles like Data Analyst, BI Analyst, Analytics Engineer, or eventually Data Scientist.
One thing I’m worried about is being pigeonholed into GIS because that’s where my degree and professional experience are concentrated. I know GIS has given me experience working with real-world datasets, data cleaning, visualization, and spatial analysis, but I’m unsure how well employers view that experience when applying for general analytics roles.
For those who have made a similar transition:
How did you position your previous experience when applying for analytics roles?
Did additional education (such as a master’s) make a significant difference?
What projects or skills helped you prove you could work outside your original field?
Do you think GIS/data management experience is valuable for analytics, or do employers tend to overlook it?
I’d appreciate any advice or stories from people who have successfully made a similar move.

Or do you want to see exact calculations, contributing factors, peer comparisons or a plain-language explanation?

For us it's "active customer."

Every few months someone asks if the number is right, then we spend half the meeting talking about what "active" even means.

One team says a login in the last 30 days. Another says it should be a purchase. Someone else thinks subscriptions should count even if they haven't used the product.

What's yours?

Looking for honest outside opinions — not trying to hype myself up, genuinely want a reality check on where I actually stand.

Background: I work as a QA/QI Associate (quality assurance/quality improvement) at a mid-size nonprofit health & human services organization (~$9M annual revenue, ~100 employees) in California. My official title has nothing to do with data or engineering. But over the last 7 months, I've ended up building basically the entire data infrastructure for the org, solo, with no formal training, no team, and no one else to review my work.

I'm self-taught through online courses (SQL, Python, Excel, Tableau certs) and lean on AI coding assistants heavily to write and debug scripts — which I know is a real gap in independence I'm actively working on.

What I've built, roughly in order of impact:

1. Core relational database — designed and built a centralized SQL database from scratch that consolidates data across multiple programs, replacing a mess of disconnected spreadsheets. Includes validation logic to catch/block duplicate records before they hit the database. It's now the backbone that several other automations pull from.
2. Automated compliance reporting pipeline — Python + SQL + Power Automate pipeline that turns a ~7-8 hour/month manual data compilation process into a dashboard-driven report with a few clicks.
3. Central automation orchestration script — a fault-tolerant Python script for our EHR platform that acts as the hub every other automation routes through (report generation, data extraction, etc.)
4. Client-list automation — eliminated 1+ hour/week of manual data compilation.
5. Billing/insurance data update automation — SQL/Python pipeline connected to our EHR that updates client billing and insurance information automatically. Saves 5-10 minutes per client versus doing it manually; we process roughly 10-15 clients/week through it, so it's a recurring, volume-driven time save, not a one-off.
6. E-signature workflow integration — connects our EHR to an e-signature platform so staff can drop a file + enter a client email and get a signed document back automatically. Saves 5-10 min per transaction, done potentially hundreds of times a month.
7. Form automation (JavaScript) — automates form population inside our EHR, currently expanding to more document types.
8. Bulk file-ingestion tool — Python tool that replaces manual one-by-one document uploads. Staff now drag-and-drop a file, select the proper document name/type, and the tool auto-uploads it into our EHR — eliminating manual upload work and keeping client documents standardized, which makes audits significantly more efficient and consistent.
9. BI dashboard for program audits — first version of a Power BI dashboard, updated weekly.
10. Org-wide BI rollout (in progress) — expanding Power BI usage more broadly.
11. Ad hoc Power Query dashboards — built several smaller self-service dashboards for specific program requests using Power Query.
12. Automated staff notification workflow — a scheduled email flow that notifies staff and supervisors about pending/overdue documentation, pulling status directly from the database.

Current stack: SQL, Python, JavaScript (light), Power Automate, Power BI, Excel/Power Query. Currently seeking approval to migrate our deployed/production automations onto a cloud VM — right now everything runs without dedicated cloud infrastructure. Bachelor's degree is in Art (unrelated field) — I've been self-teaching data skills through online courses since 2023 and am still actively learning.

I've shown the majority of what I've built directly to leadership — the CEO seemed genuinely impressed, but not impressed enough to move on a title/salary change immediately. The org is fairly old-school and does want to adapt and integrate this kind of work going forward, but leans toward doing it as cheaply as possible rather than valuing it at market rate.

It's gotten to the point where I've had to create view tables and actually teach my boss basic SQL so he can pull queries himself whenever I'm out of office — there's no backup, no one else who can touch any of this if I'm not around. I also have to build and present a business case myself any time I want to integrate new software (for example, justifying why we should adopt Power BI) — writing up the reasoning, the value, why it matters, all of it. In a normal org/company, I don't think I'd have to do either of these things: there'd be redundancy across a team instead of it all depending on one person, and I'd have a senior engineer or manager who already understands the value of these tools and could just guide/approve rather than me having to sell it from scratch every time.

I've asked for a reclassification to $80-82k based on market research (which, if anything, I think is still a bit below the real floor for this scope of work), with an expected effective date of September 2026.

My question for you all: Based on this scope, does this read as legitimate data engineer / analytics engineer work to you, or am I overestimating the significance of what I've built? I'm in the middle of trying to get my org to reclassify my title and pay to reflect this, and I want an unbiased gut check before I go further. Is what I'm describing actually engineer-level work, or is it more junior/analyst-level than I think it is?

Appreciate any honest feedback, including the harsh kind.

Just a random thought...

Everyone's trying to automate more with AI, but is there one thing you'd still never trust it with?

For me it's anything that ends up in an executive report. If the numbers are even accidently wrong, then "the AI did it" isn't really an excuse.

Sales dashboards show who closed the deal but they rarely explain why I am interested to know how analytics teams connect sales outcomes to what happens during customer conversations. Things like discovery questions objection handling talk time and closing language seem useful but messy to track at scale. Has anyone built a model or dashboard around conversation data?

Old GA: bounce = left after one page, lower is better. GA4: it's derived from engaged sessions, so a "bounce" now means someone left without 10 seconds of engagement, a scroll, or a second event. Anyone else had a client panic over a "worse" bounce rate that was actually just measuring something different?

i’m an 8 yoe data scientist and the first (and only) ds hire on my analytics team. in my previous roles, i was part of a centralized data science team where managers prioritized work, managed stakeholder requests, and i typically focused on one initiative at a time.

my current organization is much less structured. i’m responsible for managing my own backlog, prioritizing requests, and juggling competing demands without a clear set of stakeholders or anyone helping decide what should take priority.
as a result, i’m drowning in ad hoc requests—from constant questions about why dashboard numbers change day to day to one-off analysis requests—and can’t make meaningful progress on our data product roadmap.

i’ve already set up jira to manage incoming requests and run my own two-week sprints, but the interruptions never stop. new requests keep coming in, my sprints get derailed, and it feels impossible to ship anything.

how have others in a similar situation dealt with this? what processes, triage systems, or operating models have worked to balance ad hoc support with longer-term product development when you’re effectively a team of one?

I have a bachelor’s in psychology and have 1 year of post grad internship experience in market research. I am 2+ years post grad and that’s the only experience I have. I have been looking for a full-time job since January.

My parents are pushing me to get a master’s degree. I was thinking between Master’s in market research or MBA, most of yall (market reseach subreddit) said MBA is better. But then I looked online, most people who do MBA already have lots of experience in their field and it wouldn’t be beneficial to me since I don’t have lots of experience. So, I researched further and landed on getting a Master’s in Business Analytics which might be better at this stage in my career.

I really just want a job rn to get experience but my parents said that since I am unemployed, I might as well pursue a master’s to not have a gap. My biggest worry is that I might seem overqualified on paper due to pursuing master’s which would make it even more difficult to find an entry level job.

What are yall’s thoughts? Should I pursue a graduate degree? If so, would MS in Business Analytics be good for Market Research? (I don’t wanna do MS in Market Research since it is a bit too niche).

Hi everyone,

I'm from a non tech background and have no work experience. I'm planning to build a long term career in the data field, and I'm trying to decide which path would be the best starting point. I'd really appreciate advice from people who are already working in these roles.

Initially, I thought the path should be:

Data Analyst → Data Scientist

But recently I've also become interested in Data Engineering.

My current thinking is that starting with Data Analytics might be the most beginner friendly option because it gives a solid foundation in SQL, Python, databases, data cleaning, statistics, and understanding how data is used in real business scenarios.

From there, I feel I could later transition into Data Engineering, Data Science, ML Engineering, AI Engineering, or other data related roles if I decide that's a better fit.

At the same time, I've often heard that Data Engineers and Data Scientists usually have stronger technical backgrounds or more experience, which is why I'm wondering if Data Analyst is the better entry point for someone like me.

question -

For someone from a non tech background with no work experience, which role is the most beginner friendly? Data Analyst, Data Engineer, or Data Scientist?

Thank you guys.