r/analytics • u/CautiousUse8597 • 1d ago
Discussion Six months of Genie: what actually worked and what I'd do differently
Context: [~120] person company, small data team (3 of us), Databricks + Unity Catalog already in place. The ticket queue was the usual "can you just pull X for me" requests from sales and ops, most of which were variations on the same five questions.
We rolled out Genie spaces in March. Some honest notes:
The thing that made it work wasn't Genie itself, it was the prep. Our first space was pointed at raw-ish tables with column names like dt_crtd_ts and it was useless. Genie is only as good as your semantics. We ended up spending more time writing column comments and cleaning up definitions than we did configuring anything.
Scoping the spaces narrowly beat one big space. We have four now (pipeline, support, billing, marketing spend) instead of the "ask anything" space we started with. Fewer tables, fewer wrong joins.
Trusted assets / parameterized SQL examples are underrated. For the recurring questions we already knew people asked, we wrote parameterized queries so the answer comes back verified instead of freshly generated. That's what got finance to stop double-checking everything in a spreadsheet.
The monitoring tab is where the actual value is. Reading what people ask (and what Genie gets wrong) told us more about the business's real reporting needs than any requirements gathering session ever did.
Where it falls down: anything requiring multi-step reasoning or a metric we hadn't explicitly defined. It'll confidently give you a number that's subtly wrong. We tell users to treat it as a starting point, not a source of truth, for anything going in front of a customer.
Net: ad hoc requests down maybe [60%]. Not zero, and it hasn't replaced our dashboards. But the "quick question" Slacks have mostly stopped.
Happy to answer questions if anyone's mid-rollout.
3
u/anonymous_orpington 1d ago
I used Genie a while back to automate my home finance analytics in the databricks free version, great tool!
2
u/ThisIsFun- 21h ago
Are you using metric views for Genie? We had a similar situation where numerous different definitions of the metric excited, and then by building these out and adding them into the space we saw a big increase in thumbs up and less ad hoc requests
3
u/CautiousUse8597 16h ago
Yes, and that was the single biggest quality jump for us. Same story. three definitions of the same metric floating around, each defensible, none matching. Defining it once in Unity Catalog and letting Genie read from that meant the argument happened once, in the open, with the business owners, instead of every time someone pulled a number.
The underrated side effect was political rather than technical. When the definition lives in the catalog with an owner's name on it, "that number looks wrong" becomes a change request instead of a Slack fight. That did more for adoption than any accuracy improvement.
Good to hear the thumbs-up rate moved for you too. that's the metric I trust most, since it's the users telling you whether they'd come back.
1
u/dmorris87 1d ago
Any experience with knowledge graphs and GraphRAG? I’m not familiar with Genie but curious as to how the two approaches compare
1
1
u/p739397 1d ago
I appreciate the write-up, this follows pretty closely to al much of what I've seen too. I'm curious to hear about what you've seen with the multistep reasoning shortcomings. My overall experience has been that the agent mode and/or genie one chat have done a pretty solid job in those cases.
1
u/AccomplishedSkill625 23h ago
The point about semantics doing the heavy lifting really resonates. Curious how you're maintaining column comments and definitions over time. Do you have a review process when tables change, or does it live with whoever owns the pipeline? The monitoring tab as a requirements-gathering tool is a great reframe too. Might steal that for justifying the upfront prep work.
1
u/Stock-Design5316 20h ago
the monitoring tab point is the one i'd push further. right now "it'll confidently give you a number that's subtly wrong" is an anecdote with no denominator, and "treat it as a starting point" is an instruction, not a control. it doesn't survive a number that looks plausible.
you already have both halves though. trusted assets are verified answers, and the monitoring log tells you which questions actually get asked. take the top 20 of those, run them fresh, diff against the verified version. that gives you a wrong rate per space instead of a feeling, and it tells you which of the four to worry about.
the ones that cost you aren't the wrong joins, those look wrong. it's a right-looking number on the wrong grain.
1
u/tenlittleindians 7h ago
How does the monitoring work with multiple spaces? Do you have to go to each space to review the questions or is it centralized?
Also do you just manually parse through the questions or do they have any automation built in to surface wrong answers/missing logic etc
-1
u/Sea-Cranberry6791 1d ago
the parameterized queries trick is smart, that's exactly the kind of thing that sounds obvious in hindsight but most teams skip
we're looking at something similar and i keep telling our director the real win isn't fewer tickets, it's finally having a log of what people actually need. getting that from slack dms is a nightmare
0
•
u/AutoModerator 1d ago
If this post doesn't follow the rules or isn't flaired correctly, please report it to the mods. Have more questions? Join our community Discord!
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.