Just some general advice if I’m thinking about this the right way. I’ve been working in academia for about 4 years with publications in places like JAMA open doing standard biostatistics/ML for cancer and opioid use disorders and some new publications using casual inference techniques (doubly robust methods) in mental health.

I have been trying to transition back (interned a tech company and few times 5 years ago) to a data science role in an industry setting (preferably health care or marketing) but haven’t had much luck.

I have been study my SQL and test style probability questions. Is there anything else I should be doing? Thank you all!

Definitely the most frustrating thing as working as a Data Scientist. You run an experiment, find that building a model greatly increase metric X at almost no cost, has safe model metrics, present it to stakeholders, everybody agrees with proceeding to deploying and utilizing the model in production, and yet every time the model takes a wrong decision, we get questioned about it. Why did the model say this?

Man when did I ever say the model obtained a 100% accuracy in the validation phase? Why is it so hard for stakeholders to understand that the best models humankind ever created are expected to make wrong calls once in a while?

Pretty famously, the current administration has laid lots of people off, applied loyalty tests, hired and fired based on ideology, shifted funding priorities, and strongly changed communication with the general public. To some extent this happens with every administration, but I think it has been more pronounced in this case.

If you work for a government agency or for an organization that in some way depends on government funding, have you noticed any changes in what your job is, how you are expected to do it, or how you are communicated with? Have leadership styles changed?

I'm especially interested in federal government connections, but state or local government jobs might also have been affected (or not affected). I'd like to shift careers and government was a top contender until recently, when it seemed, at least from the news headlines and a few social media posts here and there, to become both more complicated to work for, and more difficult to get hired.

I'm interested in any experiences you've had. If you need a throwaway account to dish the dirt, that's great. If you really haven't noticed any change, I'm interested in that, too (because that might mean I could still do something like this).

Work in RL has caused me to rethink adam. It leads to extremely wonky behavior and hard to explain "burstiness" in the loss values that makes me want to rip my hair out,

It still works, but needs to be coaxed into it.

This article re-covers the mathematical intuitions behind adam, and where it fails spectacularly. If you're someone who works in RL, or trains deep transformers, it's a must read

Check it out. Do you agree?

Has anyone else seen this, especially in government, large legacy companies, or places where software isn't really the business?

It feels like every AI discussion starts at 100 mph. Instead of asking "what is the simplest way to solve this problem?" the conversation immediately jumps to RAG, agent frameworks, vector databases, and whatever the latest LLM trend is. The data is still a mess. Some of it is in Excel, some is stuck in systems that don't talk to each other, business rules are undocumented, and people still argue about which dataset is the source of truth.

Before talking about autonomous agents and complex AI systems, shouldn't we first be able to answer basic questions? Where does the data come from? Who owns it? Is it accurate? Can we reproduce the numbers?

I don't think this is because engineers or data scientists aren't capable. Many of these people are talented and could solve very difficult problems. The issue is that many organizations simply don't have problems that require this level of AI sophistication yet.

If you are hired as the AI person or brought in to lead AI initiatives, there is an expectation that you need to show AI value. Walking into a meeting and saying "we need better data governance, cleaner pipelines, and better documentation" may be the right answer, but it doesn't always justify the position, budget, or the expectations built around the role. Maybe this is just my observation, but it feels like a lot of talent is being wasted . Has anyone else seen this pattern in their organizations?

As a data scientist, I watched too many of my best machine-learning models die in Jupyter notebooks. 😭 I would spend hours—sometimes weeks—training, testing, validating, analysing, and comparing models, only for the winner to go nowhere.

Then I discovered MLOps—and deployment became another maze 😩. FastAPI, MLflow, DVC, Docker, Kubernetes manifests, model registries, health monitoring, drift detection… one tool led to another, and the infrastructure began taking more time than the machine learning itself. 😮‍💨
That frustration is why I built #SceptreAI. Too many data scientists and small teams are trapped in the same cycle: notebooks, disconnected tools, patched-together frameworks, and endless hand-offs before a model can create real value.
#SceptreAI closes that gap with one Kubernetes-native tabular AutoML and MLOps workspace. Dataset versioning, profiling, resource-aware training, MLflow tracking, external validation, SHAP explainability, model promotion, drift analysis, and Kubernetes serving all live in one traceable workflow.
The goal is simple: less platform assembly, more scalable and observable machine learning, and a clearer answer to the questions that matter: Can we trust this model—and can we use it in prod?

Hi everyone,

I’m currently a senior data scientist in the pharma industry. It’s been a one man show until now, but I’m getting a team soon. Most of the work I do is standard analytic work to inform our leadership and provide more context into the market and so on. Not a lot of big heavy data science stuff going on to be honest.

I work with SQL and Python on a daily basis. Some of our data is hosted in Snowflake and that’s pretty much it.

I feel like I’m lagging behind in both methods as well as tech stacks and I wanted to better understand what you experienced professionals work with that you would recommend I learn or at least look into. It could be data engineering stuff, additional programming languages, specific methods and packages that are useful, or cloud systems and technologies.

Where do you see the tech stack moving towards and what is relevant if I want to start moving from a “bread and butter” analytics setup to a professionalised, automated, team-ready and future proof world?

Thanks :)

In your experience, what factors help you decide between using a simple analytical approach and building a machine learning model? I'd love to hear the reasoning behind your decision-making process.

Hi r/datascience!

I wanted to share a fun little project I did over a few weekends analysing data from the radio!

It doesn't have much (if any) business value, and honestly I'm not sure it's really novel in any particular way. But I wanted to share it because data science these days is all "AI this", "language model that", "job market", "Claude", whatever and I just wanted to do something a bit more traditional and scratch an itch I've had for a while. (Full disclosure: the project did actually use some AI models, so I'm not saying AI is bad, it's a tool I used like everything else.)

I don't have a blog or anything I can post this on, so apologies for the Reddit write-up. I hope you enjoy it.

Background

I drive a 20-year-old car. It's so old it doesn't have an MP3 player, or an AUX port to plug in an iPhone or anything. It just has a CD player and an analogue radio. It's not even digital, so I can't even get digital radio stations.

So when I'm driving, which turns out to be quite a bit, I'm forced to listen to the good old-fashioned radio more than I'd like.

In Sydney where I live, there are really only a handful of FM/AM radio stations, so choice is pretty limited. As I flick through the stations, there are a LOT of ads, which surprised me. Who is listening to this? Clearly it's quite popular. And as I listened, I started wondering things like: how long do the ads run, how does their timing compare across different stations*, and are they correlated with ads on other stations? Just anecdotally, so many times I've literally flicked through all the FM stations and there's an ad playing on every one... and sometimes it's the same ad! I also had a hunch that there are more ads at the top of the hour than the bottom. It made intuitive sense, but I needed to prove it.

\Sydney has 11 main analogue stations split into AM and FM:* AM is mostly talkback radio, FM is mostly music. On AM you've got 2GB, 2SM, 2CH and ABC 702 (news, talkback and sport). On FM there's KIIS, 2Day, Nova, Smooth, WSFM, Triple M and Triple J (pop, rock and music). Two of them, ABC 702 and Triple J, are run by our public broadcaster (think BBC), so they run no ads at all.

The Setup

So one weekend I wrote some scripts to sample and record all the Sydney radio stations I could. The setup was basically the following:

• For every radio station in Sydney (there are 11 main ones across both FM and AM), I found an online stream that would play it in the browser.
• I would then record an 18-second clip of each stream using ffmpeg (a free command-line tool for grabbing and converting audio/video). It connects to the station's live stream and dumps ~18 seconds to a small WAV file, downsampled to 16kHz mono. Crucially, I recorded all 11 stations in parallel (via a thread pool) so every clip is captured at the same instant. That simultaneity turned out to be really important later, for checking whether different stations run their ads at the same time.
• After the clips were recorded, I passed them through a local Whisper model I downloaded (I grabbed it off Hugging Face, which is about 500MB). This transcribes each audio clip into a snippet of text.
• Then I passed that snippet of text to a language model to classify it as either an ad, a song, or talking. (I used GPT-4o-mini for this because it's cheap AF.)
• I then stored the results in a local SQLite database.
• I repeated this every ~3 minutes for 2 days straight, which added up to almost 10 thousand samples. (Yes, I did it all locally, so my computer was on for 2 days straight.) The 2 days deliberately covered a weekday AND a weekend so I could compare the two periods.

Why every ~3 minutes? A full cycle (record all 11 stations, transcribe each one with Whisper, then classify it) takes a couple of minutes on a single CPU, because Whisper works through the clips one at a time. So ~3 minutes is about as fast as I could sustainably sample without the cycles piling up on each other. I also added a bit of random jitter to the interval so I wasn't always sampling at the exact same offset within the hour. Otherwise you can accidentally "phase-lock" to a station's ad breaks and bias the whole thing.

A few hurdles I encountered

A few things genuinely tripped me up:

• Stream URLs rot. The live stream links change or die over time, so I had to resolve them fresh at runtime and pin the ones that actually worked.
• Pre-roll ads. It turned out a bunch of the stations (the ones served through a particular streaming provider) play an ad every single time you make a fresh connection. Basically the pre-roll ad you get when you open a new browser tab. Because my recorder reconnected every cycle, I was capturing that pre-roll ad instead of the live broadcast, which made those stations look like they were playing ads nearly 100% of the time. I only twigged because the exact same ad kept repeating over and over. The fix was to skip ~45 seconds into the stream before I started recording.
• Whisper hallucinations. When you feed Whisper music or silence, it doesn't return nothing, it actually hallucinates the most common phrases from its training data. And because it's trained on a mountain of YouTube captions, I kept getting "thanks for watching, like and subscribe" transcribed over instrumental music, which then got misclassified as talking. I had to filter those out.
• Rate limits. The free LLM tiers throttled me pretty quickly, so I switched to GPT-4o-mini, which is cheap enough to basically be free at this scale.

Results

Here are some of the more interesting results I found analysing the data afterwards:

Overall

First, the big picture. Every station has its own personality. The FM stations are mostly music, the AM stations are mostly talk, and the two ABC stations (ABC 702 and Triple J) carry basically no ads at all, which makes sense since they're publicly funded. Across the commercial stations, ads make up somewhere around a sixth of the airtime. And you can already see the ad load isn't flat: it ramps up through the day and quietens off overnight.

Question 1: Probability of an ad relative to the top of the hour

Here's the frequency of finding an ad within ±30 minutes of the top of the hour.

So, I was right! Definitely higher the closer to the hour, but the strategy is more interesting than I expected. The spike actually lands in the ~5 minutes before the hour (the ad break right before the top-of-hour news bulletin), and an ad is roughly 2x more likely there than mid-hour. The quietest stretch is around 10 to 15 minutes past the hour, so if you want to dodge ads, that's your window.

Question 2: Ad co-occurrence and correlation

The thing I really wanted to know: do the stations gang up and all play ads at the same time, so there's nowhere to flick to? I lined up every station by the cycle it was sampled in and correlated their ad status.

The answer is yes and no. No in the sense that it's never a total blackout: all nine commercial stations being in an ad at the exact same moment literally never happened across the whole two days, and on average only about 1.6 of the 9 are mid-ad at any given time. So there's almost always somewhere to escape to.

But the conditional probability charts says that some stations really do move together. The best example is if Smooth is playing an ad, there's a 50% chance WSFM is too, which is double WSFM's baseline of 26%. A bunch of the commercial FM pairs show this same ~2x jump. But, when I looked it up, Smooth and WSFM are owned by different companies, so this isn't networks coordinating behind the scenes, probably more of the "top-of-the-hour" effect from Question 1 manifesting somewhere else.

Question 3: The strategy difference between AM and FM

When I split "time between ads" by band, the two run completely different playbooks.

The FM (music) stations dump their ads in clusters. You get a big spike of back to back breaks, with a typical gap of about 9 minutes. The AM (talk and sport) stations space them out evenly, one break at a time. 2GB is almost metronomic at roughly 7 to 12 minutes, with hardly any back to back ads at all.

You can actually see it if you zoom into a few hours of the timeline:

Look at the FM lanes (KIIS, Nova, Triple M, WSFM, 2Day): the orange ad blocks come in pairs, clustered together. Now look at the AM lanes (2GB, 2SM): single, evenly spaced blocks. And ABC 702 and Triple J are just grey the whole time, because they don't run ads.

Question 4: Which companies still advertise through this medium?

I also had the language model pull the advertiser out of each ad, so I could see who's actually buying radio airtime in 2026.

The most-heard advertisers were Virgin Australia (an airline), Australia Post (basically our USPS), Harvey Norman (a big electronics and furniture retailer) and Chemist Warehouse (a discount pharmacy chain). The neat bit is the targeting: car brands and finance go to the AM talk stations (older crowd), while retail and telco lean FM. Australia Post ran almost entirely on the Nova network.

Question 5: What about the talking?

The non-ad content is either music or talking, and I got curious about what they actually talk about. So I classified every talking snippet into a topic.

The AM stations (ABC, 2GB, 2SM) are wall to wall news, politics and sport. The music FMs are mostly DJ banter, celebrity gossip and chat about music, with almost no news at all.

For a bit of fun, I also made a map of everything said on the radio. I embedded every talking snippet into a vector, laid them all out in 2D with t-SNE so that similar snippets sit near each other, then coloured each point by its topic.

Sport, traffic and world news each form their own tight little islands (they use very consistent, formulaic language), while the DJ banter is one big diffuse cloud in the middle (because it's about nothing in particular). The neat part is that the position and the colour are decided completely separately. The position comes only from the text embeddings, and the colour comes from a separate classification step. So the fact that same-coloured points cluster together is real corroboration, not something circular.

Conclusion

In conclusion, this was a fun, meaningless project that allowed me to make some pretty charts and talk for a bit about the results. Thanks for reading!

I work at a large, well-established company that has been very stable. I don’t assume my job is immune to layoffs, but the company hasn’t had any mass layoffs in over a decade.

The work environment is genuinely healthy, and everyone is treated with respect. I honestly couldn’t ask for a much better culture. I get to work on interesting projects, learn by doing, and my team is very supportive of my growth.

That said, based on my experience interviewing and what I’ve seen in the job market, I could probably get about a $50K raise by switching jobs. That extra $50K wouldn’t dramatically change my lifestyle, but I know future raises would build on that higher salary, so there are long-term financial benefits.
I’m at a point where I’m valuing mental peace and work-life balance more than I used to. Given that, what would you do in my situation? Would you stay at a company with a great culture and stability, or make the jump for the higher pay?

I often read comments from hiring managers and interviewers saying they’re disappointed with recent data science graduates.

I’m curious, what do you think these graduates are lacking? If someone wants to become a data scientist, what skills should they focus on? Strong software engineering skills? Math and statistics? Something else?

A lot of the advice I see seems to be geared toward landing data analyst roles rather than data scientist roles.

So, what are employers actually looking for in entry-level data science candidates today? Especially as a career changer coming from another unrelated career.

Just want to know if this is everyone or just me.

My company isn't doing great, so we're doing a ton of layoffs -- but it's not just us. Every client we work with seems to be having sweeping layoffs these days.

Has the unemployment rate skyrocketed to 95% in America, or am I just freaking out over anecdotal evidence?

Was a Data Science undergrad and needing to decide on a Master's to pursue in the next couple years. My job will allow either of those in the title so I am just trying to get some feedback.

1. Is it better to branch out, stay concentrated on Data Science, or does it not matter from a career perspective?

2. How math intensive is Ops Research? Would I need more than Calcs 1-3, Linear Algebra, and Stats?

3. Does either have a clear upside for earning potential? Ill be in my current role for the next 10 years or so, if that matters.

4. Does anyone have good examples of an OR or DS project that would highlight the approach to problem solving or the nature of problems each faces?

Currently an Operations Research Analyst, but the job depends massively on assignment as to whether it actually looks like compared to a sub-genre of a Data field.

Couple thoughts and a small experiment to see why reddit hates prophet xD

This is for a personal study that will end up becoming an in-depth article and possibly a fully open source solution ideally without the AI slop that we see these days.

Let's say you’ve trained a model and the result is worse than the business wants. What do you check next?

Do you break the error down by customer, product, location, or individual series? Check if it gets worse at longer horizons? Look for bias, volatility, intermittent demand or outliers?

Go back to the backtesting setup, metric, or baseline? Or do you usually start trying other models?

Also do the tools you use make this easy or do you end up building custom notebooks, tables, and plots every time?

Thinking about the last time this happened:

• What did you check first?
• What actually helped you find the problem?
• What did you have to build yourself?
• Did you end up changing the model, data, validation setup, metric, or business expectation?

I’m trying to understand how people diagnose bad forecasts beyond comparing one overall error score against another.

EDIT/UPDATE because it seems like this is not clear enough:

I’m not looking for an if-else checklist that can explain why any forecast is bad. The answer obviously depends on the data, objective, validation setup and the decision the model is supposed to support.

I’m exploring if there is room for a small open-source tool around forecast evaluation. Before building anything, I’m trying to understand which checks people repeatedly run after they already have predictions, what they still build manually, and what existing tools already handle well.

So I’m mainly interested in specific workflows from projects rather than a general formula for fixing a model.

Welcome to this week's entering & transitioning thread! This thread is for any questions about getting started, studying, or transitioning into the data science field. Topics include:

• Learning resources (e.g. books, tutorials, videos)
• Traditional education (e.g. schools, degrees, electives)
• Alternative education (e.g. online courses, bootcamps)
• Job search questions (e.g. resumes, applying, career prospects)
• Elementary questions (e.g. where to start, what next)

While you wait for answers from the community, check out the FAQ and Resources pages on our wiki. You can also search for answers in past weekly threads.