FreeIPA - I’ve been running FreeIPA for quite some time now on a Rocky Linux 8.9 server. Never had any major issues with it so far. That until last night when I’ve performed a full packages upgrade on server (ipa included, all packages). After upgrade, all IPA packages were version 4.9.12-11.module+el8.9.0+1652+4ee71f6a.

After that, no more functional IPA. Checking the logs I was getting:

In /var/log/krb5kdc.log:

Jan 23 23:59:38 ipa02.**.prod krb5kdc[1647](info): TGS_REQ (6 etypes {aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19), aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17), camellia256-cts-cmac(26), camellia128-cts-cmac(25)}) 10.**.**.**: ISSUE: authtime 1706052977, etypes {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha1-96(18), ses=aes256-cts-hmac-sha1-96(18)}, admin@**.PROD for krbtgt/**.PROD@**.PROD
Jan 23 23:59:38 ipa02.**.prod krb5kdc[1647](info): closing down fd 12

In /var/log/httpd/error_log:

[Tue Jan 23 21:57:55.535671 2024] [wsgi:error] [pid 18724:tid 139814201841408] [remote 10.**.**.**:51948] ipa: INFO: 401 Unauthorized: Insufficient access: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Credential cache is empty)
[Tue Jan 23 21:57:55.644487 2024] [wsgi:error] [pid 18722:tid 139814201841408] [remote 10.**.**.**:51948] ipa: INFO: 401 Unauthorized: Insufficient access: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Credential cache is empty)

No IPA cli commands were working, I was not able to login into the IPA Web UI. The only thing that was actually working was:

kinit admin

Password was accepted but that was pretty much it.

So after about 4 hours or reading and trying to fix it based on whatever solutions I was able to find online, old and new, the conclusion was pretty clear. I had to enable somehow SIDs for all users.

So I tried:

ipa config-mod --enable-sid --add-sids

Unfortunately, that didn’t work.

Tried to downgrade all IPA packages but there was some IPA database version mismatch which prevented IPA to start so I had to re-upgrade to the latest version of the packages.

I tried to create a full IPA backup then I remembered that I had a server snapshot from 3 months ago so it started to look good. My thinking was: Create a new VM from that snapshot, start it with a different ip, restore the IPA backup and you’re good to go. Well easier said than done since again some version mismatch!

In the end, what worked for me and fixed my FreeIPA was this:

/usr/libexec/ipa/oddjob/org.freeipa.server.config-enable-sid \
    --netbios-name EXAMPLE \
    --add-sids

Here’s the full FreeIPA thread.

So even RedHat can screw it up! Make regular backups, DYOR before upgrading!

LE: I just got a reply from a RedHat Software Engineer with the official solution, so I’m posting it here as well:

https://access.redhat.com/solutions/7052703.

It requires a subscription but it works with a free developer subscription.