Securing Ozone OM

[Dalamar32]

I am trying to setup kerberos security in order to secure an existing HA cluster and I am running into issues after following the documentation on securing ozone.

After setting the properties needed for kerberos to work with ozone as per the documentation, OM fails to start up citing:

java.lang.IllegalArgumentException: Can't get Kerberos realm

java.lang.IllegalArgumentException: KrbException: krb5.conf loading failed

Even with the krb5.conf setup including the default_realm variable.

I am wondering if it can't find the krb5.conf file. Is there a way to specify the location of the krb5.conf in the ozone-site.xml?

[fapifta]

Location of krb5.conf file should be either by default the /etc/krb5.conf, or it should be in the place defined by the KRB5_CONFIG environment variable.
If you can share the full exception trace that should help to better understand what might be the problem, but in general the underlying facilities are coming from Hadoop via its UserGroupInformation class, and related logic/code around Ugi.

Also you may enable kerberos related debug logging in the jvm (-Dsun.security.krb5.debug=true in OZONE_OPTS env var should do it), and see if the kerberos layer gives more info on what is missing.

Based on what you have shared from the exception itself, I tend to believe that there is a syntax problem within the krb5.conf file.

[Dalamar32]

Here is the full trace:

2023-06-12 12:13:40,882 [main] ERROR org.apache.hadoop.ozone.om.OzoneManagerStarter: OM start failed with exception java.lang.IllegalArgumentException: Can't get Kerberos realm at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:71) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:315) at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:366) at org.apache.hadoop.ozone.om.OzoneManager.loginOMUser(OzoneManager.java:1058) at org.apache.hadoop.ozone.om.OzoneManager.loginOMUserIfSecurityEnabled(OzoneManager.java:1212) at org.apache.hadoop.ozone.om.OzoneManager.<init>(OzoneManager.java:473) at org.apache.hadoop.ozone.om.OzoneManager.createOm(OzoneManager.java:677) at org.apache.hadoop.ozone.om.OzoneManagerStarter$OMStarterHelper.start(OzoneManagerStarter.java:189) at org.apache.hadoop.ozone.om.OzoneManagerStarter.startOm(OzoneManagerStarter.java:86) at org.apache.hadoop.ozone.om.OzoneManagerStarter.call(OzoneManagerStarter.java:74) at org.apache.hadoop.hdds.cli.GenericCli.call(GenericCli.java:38) at picocli.CommandLine.executeUserObject(CommandLine.java:1953) at picocli.CommandLine.access$1300(CommandLine.java:145) at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352) at picocli.CommandLine$RunLast.handle(CommandLine.java:2346) at picocli.CommandLine$RunLast.handle(CommandLine.java:2311) at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179) at picocli.CommandLine.execute(CommandLine.java:2078) at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:100) at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:91) at org.apache.hadoop.ozone.om.OzoneManagerStarter.main(OzoneManagerStarter.java:58) Caused by: java.lang.IllegalArgumentException: KrbException: krb5.conf loading failed at java.security.jgss/javax.security.auth.kerberos.KerberosPrincipal.<init>(KerberosPrincipal.java:179) at org.apache.hadoop.security.authentication.util.KerberosUtil.getDefaultRealm(KerberosUtil.java:120) at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:69) ... 20 more 2023-06-12 12:13:40,896 [shutdown-hook-0] INFO org.apache.hadoop.ozone.om.OzoneManagerStarter: SHUTDOWN_MSG:

I am testing with debug enabled today and will update with more info soon.

[fapifta]

For me based on the exception it feels like as if the default_realm would not be correct, or it may not be specified in the krb5.conf file.

The failing piece is deep within the Java security code, where the config file is being parsed, but the concrete exception is swallowed by the KerberosPrincipal constructor, so we just get the error message to infer from.
Can you please verify if your krb5.conf is correct, and the realm is also valid?
Also you should check in the debug logs where the krb5.conf is being loaded (I believe it is logged), is that the path you expect and check?

[Dalamar32]

I verified that the krb5.conf and the default_realm is correct.

I found a new log file with the .out extension that provided more detail:

java.io.IOException: Profile path included more than once
at java.security.jgss/sun.security.krb5.Config.readConfigFileLines(Config.java:593)
at java.security.jgss/sun.security.krb5.Config.readConfigFileLines(Config.java:620)
at java.security.jgss/sun.security.krb5.Config$1.run(Config.java:690)
at java.security.jgss/sun.security.krb5.Config$1.run(Config.java:681)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:726)
at java.security.jgss/sun.security.krb5.Config.loadConfigFile(Config.java:680)
at java.security.jgss/sun.security.krb5.Config.(Config.java:232)
at java.security.jgss/sun.security.krb5.Config.getInstance(Config.java:132)
at java.security.jgss/sun.security.krb5.Realm.getDefault(Realm.java:63)
at java.security.jgss/sun.security.krb5.PrincipalName.(PrincipalName.java:468)
at java.security.jgss/sun.security.krb5.PrincipalName.(PrincipalName.java:477)
at java.security.jgss/javax.security.auth.kerberos.KerberosPrincipal.(KerberosPrincipal.java:177)
at org.apache.hadoop.security.authentication.util.KerberosUtil.getDefaultRealm(KerberosUtil.java:120)
at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:69)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:315)
at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:366)
at org.apache.hadoop.ozone.om.OzoneManager.loginOMUser(OzoneManager.java:1058)
at org.apache.hadoop.ozone.om.OzoneManager.loginOMUserIfSecurityEnabled(OzoneManager.java:1212)
at org.apache.hadoop.ozone.om.OzoneManager.(OzoneManager.java:473)
at org.apache.hadoop.ozone.om.OzoneManager.createOm(OzoneManager.java:677)
at org.apache.hadoop.ozone.om.OzoneManagerStarter$OMStarterHelper.start(OzoneManagerStarter.java:189)
at org.apache.hadoop.ozone.om.OzoneManagerStarter.startOm(OzoneManagerStarter.java:86)
at org.apache.hadoop.ozone.om.OzoneManagerStarter.call(OzoneManagerStarter.java:74)
at org.apache.hadoop.hdds.cli.GenericCli.call(GenericCli.java:38)
at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
at picocli.CommandLine.access$1300(CommandLine.java:145)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
at picocli.CommandLine.execute(CommandLine.java:2078)
at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:100)
at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:91)
at org.apache.hadoop.ozone.om.OzoneManagerStarter.main(OzoneManagerStarter.java:58)
Can't get Kerberos realm

[fapifta]

Based on the code I could find quickly for java.security.jgss/sun.security.krb5.Config, this exception can happen when you load a kerberos config file that includes an other config file via the include or includedir directive and accidentally it includes itself.

In the kerberos debug output search for lines like: Loading krb5 profile at <path> and you should see two lines with the same path, that is the only cause I see obvious from the code which throws this IOException.

[Dalamar32]

In the /etc/krb5.conf I removed the following:

includedir /etc/krb5.conf.d/
includedir /var/lib/sss/pubconf/krb5.include.d/

Afterwards OM came up successfully

[MerelyOneMax1986]

Dear @Dalamar32, it feels like you are solving the same problem as me. I would appreciate it if you we could make some kind of collaboration as Ozone is a cutting edge system and it's obvious that we'll face a lot of concerns on the way.

Currently, could you please share core-site.xml and hdfs-site.xml (and other conf) for your setup? I'm struggling with the interesting enterprise-level setup: Apache Ozone 1.3.0 + Spark 3.2.3 (cluster mode) + Kerberos. All resides in Kubernetes. And I ended up with absence of Ozone delegation tokens for Spark workers and didn't find any meaningful errors in the logs.