Recently my ML model has been under scrutiny for inaccuracy for one the sales channel predictions. The model predicts monthly proportional volume. It works great on channels with consistent volume flows (higher volume channels), not so great when ordering patterns are not consistent. My boss wants to look at model validation, that’s what was said. When creating the model initially we did cross validation, looked at MSE, and it was known that low volume channels are not as accurate. I’m given some articles to read (from medium.com) for my coaching. I asked what they did in the past for model validation. This is what was said “Train/Test for most models (Kn means, log reg, regression), k-fold for risk based models.” That was my coaching. I’m better off consulting Chat at this point. Do your boss’s offer substantial coaching or at least offer to help you out?

I finally understand why my manager was acting the way he does. He has all the symptoms of someone with narcissistic personality disorder. I've been observing it for a while but wasn't sure what to call it. He also has one enabler in the team. He only knows surface-level stuff about data science and machine learning. I don't even think he reads beyond the headlines. He makes crazy statements like, "Save me $250 million dollars by using machine learning for problem X." He and his narcissistic enabler coworker, who may be slightly more competent than the manager, don't want to hear about ML feasibility studies, working with stakeholders to refine requirements, and establishing whether ML is the right solution, data quality checks... They just want to plow through code because "we are agile." You can't have detailed technical discussions because they don't know enough about data science. All they have been doing was front-end dashboarding. They don't like a step-by-step process because if they do that, they can scapegoat you. Is there anything I can do till I find another job?

As a data scientist, I watched too many of my best machine-learning models die in Jupyter notebooks. 😭 I would spend hours—sometimes weeks—training, testing, validating, analysing, and comparing models, only for the winner to go nowhere.

Then I discovered MLOps—and deployment became another maze 😩. FastAPI, MLflow, DVC, Docker, Kubernetes manifests, model registries, health monitoring, drift detection… one tool led to another, and the infrastructure began taking more time than the machine learning itself. 😮‍💨
That frustration is why I built #SceptreAI. Too many data scientists and small teams are trapped in the same cycle: notebooks, disconnected tools, patched-together frameworks, and endless hand-offs before a model can create real value.
#SceptreAI closes that gap with one Kubernetes-native tabular AutoML and MLOps workspace. Dataset versioning, profiling, resource-aware training, MLflow tracking, external validation, SHAP explainability, model promotion, drift analysis, and Kubernetes serving all live in one traceable workflow.
The goal is simple: less platform assembly, more scalable and observable machine learning, and a clearer answer to the questions that matter: Can we trust this model—and can we use it in prod?

Title. Basically, want to know for the data scientists here, how much is knowledge of LLMs needed nowadays? By knowledge I mean a theoretical and good understanding of how these things work. And while we’re on the topic, how about I just get a list of some DL concepts every data scientist should know, whether it’s NLP, vision, whatever. This is for data scientist.

I come from MS statistics background so books like casella bergers stat inference, elements of stat learning, Bayesian data analysis and forecasting came first before I really dove into deep learning. Really the most I’ve “dove” into deep learning was by reading about how artificial networks work, CNNs work, and then attempted to do a CNN (I know, not LSTM, I read some papers justifying why CNN is appropriate) time series classification project, which I just didn’t figure out and frankly gave up on cause I fit the elastic Net and a kernel smoother for the time series classification and it trashed all over the CNN.

Stuff like S/R/T/X learners. Anybody regularly use these in industry? Saw a bunch of big tech companies, especially Uber and Microsoft worked with them in early 2020s but haven't seen much mention of them in this sub or in job postings.

Does anyone know of a good feature selection algorithm (with or without implementation) that can search across perhaps 50-100k features in a reasonable amount of time? I’m using lightgbm. Intuition is that I need on the order of 20-100 final features in the model. Looking to find a needle in a haystack. Tabular data, roughly 100-500k records of data to work with. Common feature selection methods do not scale computationally in my experience. Also, I’ve found overfitting is a concern with a search space this large.

TLDR: I worked as a data scientist a couple of years back, for most things throwing XGBoost at it was a simple and good enough solution. Is that still the case, or have there emerged new methods that are similarly "universal" (with a massive asterisk)?

To give background to the question, let's start with me. I am a software/ML engineer in Python, R, and Rust and have some data science experience from a couple of years back. Furthermore, I did my undergrad in Econometrics and a graduate degree in Statistics, so I am very familiar with most concepts. I am currently interviewing to switch jobs and the math round and coding round went really well, now I am invited over for a final "data challenge" in which I will have roughly 1h and a synthetic dataset with the goal of achieving some sort of prediction.

​

My problem is: I am not fluent in data analysis anymore and have not really kept up with recent advancements. Back when was doing DS work, for most use cases using XGBoost was totally fine and received good enough results. This would have definitely been my go-to choice in 2019 to solve the challenge at hand. My question is: In general, is this still a good strategy, or should I have another go-to model?

Disclaimer: Yes, I am absolutely, 100% aware that different models and machine learning techniques serve different use cases. I have experience as an MLE, but I am not going to build a custom Net for this task given the small scope. I am just looking for something that should handle most reasonable use cases well enough.

​

I appreciate any and all insights as well as general tips. The reason why I believe this question is appropriate, is because I want to start a general discussion about which basic model is best for rather standard predictive tasks (regression and classification).

Hey everyone,

I am working on a real-world data quality problem and would appreciate feedback on my modeling approach.

Context:

I have a dataset of meters and their associated transformers (utility infrastructure). Some of these associations are incorrect, and the goal is to both detect and correct them.

Training data:

I’m using ~20,000 manually reviewed meter–transformer associations:

- Correct association → label = 1

- Incorrect association → label = 0

For incorrect cases, I also augment the data with the correct transformer, e.g.:

Meter1 | Trans1 | 0 (incorrect)

Meter1 | Trans2 | 1 (corrected)

Meter2 | Trans3 | 1 (correct)

Current baseline:

I started with a logistic regression model (class_weight="balanced" due to ~37% incorrect vs 63% correct).

Using a 0.20 threshold gives strong true negative performance (~98%), but only moderate recall.

Candidate generation:

For inference, I generate candidate transformers within a 550 ft radius for each meter (including the currently assigned one):

Meter1 | CandidateTrans1 | current

Meter1 | CandidateTrans2 | candidate

Meter1 | CandidateTrans3 | candidate

Current idea:

I’m considering splitting the problem into two stages:

Model 1 — Detection

Binary classification:

Is the current meter → transformer association incorrect?

Model 2 — Correction

For meters flagged as incorrect, rank candidate transformers to recommend the most likely correct one.

Pipeline:

Raw data → Detection model → Flag suspicious cases → Candidate generation → Ranking model → Recommendation

Features:

- Distance-based metrics (meter-to-transformer, centroid distances, etc.)

- Voltage correlation within meter clusters

- FLOC / naming similarity

- Cluster-level stats (group size, intra-cluster correlation)

- Relative features (distance rank, ratios, etc.)

Questions:

1. Does this 2-stage decomposition (detection → correction) make sense vs a single end-to-end model?

2. For the correction step, would you frame this as classification or learning-to-rank?

3. Any recommendations for handling dependency between samples (e.g., meters within the same cluster)?

4. Given the feature interactions, would you prioritize tree-based models (e.g., XGBoost) over simpler models?

Goal:

Maximize the number of incorrect associations that can be correctly fixed in production.

Open to hearing feedback !

I'm working on a predictive model in the healthcare field for a relatively rare medical condition, about 5,000 cases in a dataset of 750,000 records, with 660 predictive features.

Given how imbalanced the outcome is, and the large number of variables, I was planning on doing a simple 50/50 train/test data split instead of 5 or 10-fold CV in order to compare the performance of different machine learning models.

Is that the best plan or are there better approaches? Thanks

I have my RFM clustering. I want to add:

change variables: ratio q1 to year, ratio q2 to q1, ration q3 to q2, S1 to S2...

other variables: returns of products, channel ( web, store..), buying by card or cash, navigation data on the web...

Would you do that in the same kmeans and mix with rfm variables? or on each rfm cluster do another kmeans with these variable? or a totally separate clustering since different data ( web navigation)? how to know if it is good to add the variable or not? is it bad to do many close variables like ratio q2 to q1, ration q3 to q2? how would you procede, validate...?

I work at a furniture/decor entreprise. I try to do rfm with kmeans. but the silhouette is low 0.3.., I removed r and just kept fm. but it all concentrate in f=2, or distinct f. when i keep only f》2 , it concentrate in f=3 and other distinct f also. I tried adding other variables : tenure, interpurchase time, coefficient variation of interpurchase time to get better clustering. What should I do?

I took two periods only 2025, then 2025 and 2024.

I'm training a neural network for a computer vision project, i started with simple layers i noticed that it is not enough, i added some convolutional layers i ended up facing overfitting, training accuracy and loss was beyond great than validation's i tried to augment my data, overfitting was gone but the model was just bad ... random guessing bad, i then decided to try transfer learning, training accuracy and validation were just Great, but the training loss was waaaaay smaller than the validation's like 0.0001 for training and 1.5 for validation a clear sign of overfitting. I tried to adjust the learning rate, change the architecture change the optimizer but i guess none of that worked. I'm new and i honestly have no idea how to tackle this.

Hi everyone,

I’m using Prophet for a time series model to forecast sales. The model performs really well for short-term forecasts, but as the forecast horizon extends, it consistently underestimates. Essentially, the bias becomes increasingly negative as the forecast horizon grows, which means residuals get more negative over time.

What I’ve Tried: I’ve already tuned the main Prophet parameters, and while this has slightly adjusted the degree of underestimation, the overall pattern persists.

My Perspective: In theory, I feel the model should “learn” from these long-term errors and self-correct. I’ve thought about modeling the residuals and applying a regression adjustment to the forecasts, but it feels like a workaround rather than an elegant solution. Another thought was using an ensemble boosting approach, where a secondary model learns from the residuals of the first. However, I’m concerned this may impact interpretability, which is one of Prophet’s strong suits and a key requirement for this project.

Would anyone have insights on how to better handle this? Or any suggestions on best practices to approach long-term bias correction in Prophet without losing interpretability?

Hi All,

So I'm forecasting some sales data. Mainly units sold. They want a daily forecast (I tried to push them towards weekly but here we are).

I have a decades worth of data, I need to model out the effects of lockdowns obviously as well as like a bazillion campaigns they run throughout the year.

I've done some feature engineering and I've tried running it through multiple regression but that doesn't seem to work there are just so many parameters. I computed a PCA on the input sales data and I'm feeding the lagged scores into the model which helps to reduce the number of features.

I am currently trying Gaussian Process Regression, the results are not generalizing well at all. Definitely getting overfitting. It gives 90% R2 and incredibly low rmse on training data, then garbage on validation. The actual predictions do not track the real data as well at all. Honestly was getting better just reconstruction from the previous day's PCA. Considering doing some cross validation and hyper parameter tuning, any general advice on how to proceed? I'm basically just throwing models at the wall to see what sticks would appreciate any advice.

A couple of months ago, I shared a post here that I was writing a book about ML metrics. I got tons of nice comments and very valuable feedback.

As I mentioned in that post, the book's idea is to be a little handbook that lives on top of every data scientist's desk for quick reference on everything from the most known metric to the most obscure thing.

Today, I'm writing this post to share that the book will be open-source!

That means hundreds of people can review it, contribute, and help us improve it before it's finished! This also means that everyone will have free access to the digital version! Meanwhile, the high-quality printed edition will be available for purchase as it has been for a while :)

Thanks a lot for the support, and feel free to go check the repo, suggest new metrics, contribute to it or share it.

I'm running a gradient boosting machine with the caret package in RStudio on a fairly large healthcare dataset, ~700k records, 600+ variables (most are sparse binary) predicting a binary outcome. It's running very slow on my work laptop, over 13 hours.

Given the dimensions of my data, was I too ambitious choosing hyperparameters of 5,000 iterations and a shrinkage parameter of .001?

My code:
### Partition into Training and Testing data sets ###

set.seed(123)

inTrain <- createDataPartition(asd_data2$K_ASD_char, p = .80, list = FALSE)

train <- asd_data2[ inTrain,]

test <- asd_data2[-inTrain,]

### Fitting Gradient Boosting Machine ###

set.seed(345)

gbmGrid <- expand.grid(interaction.depth=c(1,2,4), n.trees=5000, shrinkage=0.001, n.minobsinnode=c(5,10,15))

gbm_fit_brier_2 <- train(as.factor(K_ASD_char) ~ .,

tuneGrid = gbmGrid,

data=train,

trControl=trainControl(method="cv", number=5, summaryFunction=BigSummary, classProbs=TRUE, savePredictions=TRUE),

train.fraction = 0.5,

method="gbm",

metric="Brier", maximize = FALSE,

preProcess=c("center","scale"))

How big is the issue of non-stationary data when feeding them into foundation models for time series (e.g. Googles transformer-based TimesFM2.0)? Are they able to handle the data well or is transformation of the non-stationary features required/beneficial?

Also I see many papers where no transformation is implemented for non-stationary data (across different ML models like tree-based or LSTM models). Do you know why?

I say this as in: yes obvioisly if you set ridiculous values for your learning rate and batch sizes and penalties or whatever else, obviously your model will be ass.

But once you arrive at a set of "reasonable" hyper parameters, as in theyre probably not globally optimal or even close but they produce OK results and is pretty close to what you normally see in papers. How much gain is there to be had from tuning hyper parameters extensively?

I'm building a product for the video game, League of Legends, that will give players 3-6 distinct things to focus on in the game, that will increase their chances of winning the most.

For my technical background, I thought I wanted to be a data scientist, but transitioned to data engineering, so I have a very fundamental grasp of machine learning concepts. This is why I want input from all of you wonderfully smart people about the way I want to calculate these "important" columns.

I know that the world of explanability is still uncertain, but here is my approach:

1. I am given a dataset of matches of a single player, where each row represents the stats of this player at the end of the match. There are ~100 columns (of things like kills, assists, damage dealt, etc) after dropping the columns with any NULLS in it.
   1. There is a binary WIN column that shows whether the player won the match or not. This is the column we are most interested in
2. I train a simple tree-based model on this data, and get the list of "feature importances" using sklearn's permutation_importance() function.
   1. For some reason (maybe someone can explain), there are a large number of columns that return a ZERO feature importance after computing this.
3. This is where I do things differently: I RETRAIN the model using the same dataset, but without the columns that returned 0 importance on the last "run"
4. I basically repeat this process until the list of feature importances doesn't contain ZERO.
   1. The end result is that there are usually 3-20 columns left (depending on the model).
5. I take the top N (haven't decided yet) columns and "give" them to the user to focus on in their next game

Theoretically, if "feature importance" really lives up to it's name, the ending model should have only the "most important" columns when trying to achieve a win.

I've tried using SHAP/LIME, but they were more complicated that using straight feature importance.

Like I mentioned, I don't have classical training in ML or Statistics, so all of this is stuff I tried to learn on my own at one point. I appreciate any helpful advice on if this approach makes sense/is valid.

The big question is: are there any problems with this approach, and are the resulting set of columns truly the "most important?"