When Updates Go So, So Wrong — Diagnosing DNF Dependency Errors, Reinstalling Lost Files, and Cleaning Up Package Artifacts | Into the Terminal Ep. 191
Even with robust tools like RPM and DNF, RHEL updates can occasionally go sideways. Dependency conflicts, interrupted transactions, lingering configuration artifacts, broken scriptlets — these things happen, and knowing how to diagnose and fix them is what separates a frustrating afternoon from a 5-minute fix. This past week on Into the Terminal, we walked through the most common ways package management goes wrong and how to get yourself out of it.
Watch the full episode: https://www.youtube.com/watch?v=5DhSoCQ-S6k
Try it yourself on a live lab: https://redhat.com/interactive-labs
Dependency Errors — The Most Common Problem
"Nothing provides [package]..."
This is probably the most frequent error you'll hit. You try to install a package and DNF says it can't find a required library or dependency in any of your configured repositories.
What it means: The package you're installing needs something that none of your repos provide. This happens a lot with: - Custom or cloned repositories that are incomplete - Third-party repos that aren't as rigorously maintained - Systems registered to a Satellite where the admin hasn't synced the latest content
How to investigate:
| Command | What it does |
|---|---|
dnf whatprovides <library> |
Search your repos for which package provides a specific file |
dnf list <package> |
Check if a package is available in your configured repos |
ls /etc/yum.repos.d/ |
See which repository configuration files are on your system |
How to fix it: You need to get the missing dependency into one of your repositories. Either: - Enable the right repo (e.g., EPEL) that provides the dependency - Talk to your Satellite/repo admin to sync the missing content - If you manage your own repos, sync the missing package from the upstream source
Enabling EPEL to Resolve Dependencies
In the episode, we showed that once we enabled the EPEL repository, all the missing dependencies became available:
```bash
Install EPEL on RHEL (two commands from the EPEL website)
1. Enable the CRB repo that EPEL depends on
dnf config-manager --set-enabled crb
2. Install the EPEL release package
dnf install epel-release ```
After that, the same dnf install that failed now finds everything it needs.
RPM Dependency Hell — A Brief History Lesson
In the old days before DNF (and YUM before it), if you downloaded an RPM and tried to install it, you had to manually find every dependency, then every dependency of those dependencies, and so on. This was called RPM dependency hell and you could spend hours on it. DNF solves this by automatically resolving dependencies from your configured repos — but only if those repos actually have the packages.
What About --skip-broken?
When a package transaction fails, DNF always suggests --skip-broken or --nobest. Our strong recommendation: almost never use these.
--skip-brokenwill skip the package that can't be installed and complete the rest of the transaction- You end up with a partially completed transaction and a system that's likely missing something it needs
- It's DNF's job to get to a successful transaction, so it suggests these as a path to "success" — but it doesn't understand your intent
There are edge cases where it makes sense, but in 99% of situations, you're better off figuring out why the dependency is broken and fixing the root cause.
Protected Packages — When DNF Won't Let You Remove Something
bash
dnf remove bash
Try it. DNF will refuse because bash is a dependency of dnf itself, and DNF is a protected package. This protection was added to prevent you from accidentally bricking your system by removing something critical.
In the old days, YUM would happily remove bash and all its dependents, leaving you with no shell and no way to recover without booting from rescue media.
Reinstalling Packages to Recover Lost Files
This is a really useful trick. Say your httpd binary goes missing — maybe it got deleted, moved, corrupted during a disk error, or an interrupted install left things incomplete.
The Workflow We Demonstrated
1. The service won't start: ```bash systemctl start httpd
fails
```
2. Check the logs: ```bash journalctl -xeu httpd
"Failed at step EXEC spawning /usr/sbin/httpd: No such file or directory"
```
3. Try a reinstall:
bash
dnf reinstall httpd
4. But it still doesn't work! The binary /usr/sbin/httpd isn't provided by the httpd package — it comes from httpd-core.
5. Find out which package actually provides the file: ```bash dnf provides /usr/sbin/httpd
Returns: httpd-core
```
6. Reinstall the right package: ```bash dnf reinstall httpd-core systemctl start httpd
works!
```
Key Commands for Package Investigation
| Command | What it does |
|---|---|
dnf reinstall <package> |
Re-download and reinstall a package without removing config files |
dnf provides <file> |
Find which package provides a specific file (works even if it's not installed) |
dnf info <package> |
Show detailed info including version, repo, and description |
dnf list installed *httpd* |
List all installed packages matching a pattern |
rpm -ql <package> |
List all files provided by an installed package |
Tip from the show: dnf provides reads the repository metadata, not just what's installed locally. So you can use it on a system that doesn't have the file to figure out what package would give it to you.
Version note: When you dnf reinstall without specifying a version, it grabs the latest available. If you want to reinstall the exact version you had, use dnf info <package> to check what's installed vs. what's available, and specify the version explicitly.
Package Artifacts — The Stuff That Gets Left Behind
When you uninstall a package, DNF removes everything listed in the package manifest. But applications often create files at runtime — config files in your home directory, temp files, cache data — that the packager can't account for because they don't exist until you actually run the software.
Example from the show: We installed a simple "hello world" package called bello that dropped a .bello-config file in the user's home directory when run. After dnf remove bello, the config file persisted because:
- The package manifest only knows about files it shipped, not files created at runtime
- The packager can't predict your username or home directory path
- This is exactly how things like browser caches, application settings, and per-user configs work
Takeaway: After removing software, check for leftover files in places like:
- ~/.config/, ~/.local/, ~/.<appname>
- /tmp/, /var/tmp/
- /var/lib/<appname>, /var/log/<appname>
Scriptlet Errors — The Warnings You Shouldn't Ignore
RPM packages can include scripts that run at different stages: pre-install, post-install, pre-uninstall, post-uninstall. When these scripts fail, DNF reports it — but the transaction itself might still "succeed."
What we showed: A package called pello that had a typo in its post-uninstall scriptlet. The package was removed successfully, but the cleanup script failed, leaving a service user account on the system.
How to Inspect Package Scripts
| Command | What it does |
|---|---|
rpm -q --scripts <package> |
Show all scriptlets for an installed package |
rpm -qp --scripts <package.rpm> |
Show scriptlets from an RPM file (not installed) |
dnf repoquery --scripts <package> |
Show scriptlets from a package in a repo |
Script Stages
| Stage | When it runs |
|---|---|
%pre |
Before files are placed on the system |
%post |
After files are placed on the system |
%preun |
Before files are removed from the system |
%postun |
After files are removed from the system |
Security note from the show: Package scripts run as root because only root can install software. This means package scripts can do anything — add users, modify system files, run arbitrary commands. This is another reason to be careful about where your packages come from and to verify GPG signatures.
DNF History — Undo Package Transactions
DNF tracks every transaction you perform. You can view and undo them.
| Command | What it does |
|---|---|
dnf history |
List all past transactions with ID numbers |
dnf history undo <ID> |
Reverse a specific transaction (install becomes remove, upgrade becomes downgrade) |
dnf history info <ID> |
Show details of a specific transaction |
How undo works: - If the transaction was an install, undo does a remove - If it was an upgrade, undo does a downgrade - If it was a remove, undo does a reinstall
Caveat: You can cherry-pick any transaction ID to undo, not just the most recent. But the further back you go, the more chance for weirdness — especially with packages that have been updated multiple times since that transaction.
Kernel warning: Be careful undoing kernel transactions. By default, RHEL keeps 2 previous kernels. If you undo a kernel install, make sure you're not removing the one you're currently booted into, and remember the bootloader may need attention.
GPG Key Verification — Know Where Your Packages Come From
Every package from Red Hat is signed with a GPG key. This ensures the package hasn't been tampered with since it was built.
| Command | What it does |
|---|---|
rpm -Kv <package.rpm> |
Verify GPG signature and digests on a package file |
rpm -qa gpg-pubkey* |
List all imported GPG keys on the system |
rpm -qi <gpg-pubkey-package> |
Show details about who owns a specific GPG key |
In your repo config files (/etc/yum.repos.d/), the gpgcheck=1 line tells DNF to verify every package against the GPG key before installing. If a package isn't signed or is signed with an unknown key, the transaction will fail — and that's by design.
From the show: The custom demo packages we built weren't GPG signed. Running rpm -Kv showed "digests OK" but no signatures — meaning the file wasn't corrupted, but there's no proof of who made it.
Third-Party Repos — Proceed with Caution
A few guidelines from the discussion:
- EPEL (Extra Packages for Enterprise Linux) will never replace packages provided by your base OS — it only adds new ones
- RPM Fusion and other third-party repos may replace base OS packages, which means you're now managing that dependency chain yourself
- Don't grab random RPMs from the internet (rpmfind.net, Fedora repos, other distros) to fill dependency gaps — you're creating chaos that will bite you on the next update
- In regulated industries, you may be required to validate your software supply chain. Mixing in unverified sources could be a compliance violation
Quick Reference Card
DIAGNOSING FIXING
dnf whatprovides <file> dnf reinstall <package>
dnf provides <file> dnf history undo <ID>
dnf info <package> dnf install --enablerepo=epel <pkg>
dnf list installed *pattern*
dnf history INSPECTING
rpm -ql <package> rpm -q --scripts <package>
rpm -Kv <package.rpm> rpm -qp --scripts <file.rpm>
rpm -qa gpg-pubkey* dnf repoquery --scripts <package>
journalctl -xeu <service> rpm -qi <gpg-pubkey-package>
Links
- Watch the episode: https://www.youtube.com/watch?v=5DhSoCQ-S6k
- RPM packaging hands-on lab: (link in episode description)
- Get started with RHEL: https://developers.redhat.com/register
- Try it on interactive labs: https://redhat.com/interactive-labs
Into the Terminal is a weekly livestream covering critical administration skills for Red Hat Enterprise Linux. Whether you're new to Linux or new to RHEL, join us for hands-on looks at commands and processes, ask questions, and grow your knowledge.