(no title)

Solving NFSv4 AUTH_SYS nobody ownership issue

For files/directories under NFSv4 AUTH_SYS mount, if the ownership is shown as nobody, then check NFSv4 ID Mapping settings. You need to ensure that NFsv4 ID Mapping Domain (e.g. Domain attribute in /etc/idmapd.conf on RHEL7.x) is same across NFS server and NFS client.

To illustrate it, examples are provided below for list operation (“ls” command) and file create operation (“echo xyz > file” command) with RHEL7.2 NFS client.

Example 1: details about nobody ownership issue shown under nfsv4 sys mount.
Example 2: shows how to solve the issue presented in Example 1.

(Examples talk about NFSv4 ID Map domain. This is not necessarily your host domain name but can be anything and you can set it explicitly by editing Domain attribute in /etc/idmapd.conf file on RHEL machine. The configuration file and the option would change as per the NFS client OS.)

NFS server and NFS client Setup for Example 1 and Example 2:

1. To explain nobody ownership issue, we need a user account that resolves on both NFS client and NFS server. You can do this by various ways e.g. create local users on NFS client and NFS server or configure NFS client and NFS server against common authentication source (e.g. Active Directory, LDAP).

For time being, I’ve configured NFS server and NFS client against common authentication source (Active directory here) and AD user GANESHA\aduser1 resolves successfully returning uid:gid (15001:10001) on NFS server and NFS client .

# server # id GANESHA\aduser1 uid=15001(GANESHA\aduser1) gid=10001(GANESHA\ganeshagr1) groups=10001(GANESHA\ganeshagr1),20000(GANESHA\group20k)… # client $ id GANESHA\aduser1 uid=15001(GANESHA\aduser1) gid=10001(GANESHA\ganeshagr1) groups=10001(GANESHA\ganeshagr1),20000(GANESHA\group20k)…

2. Create a file on server owned by user GANESHA\aduser1.

# create file on server and set ownership to GANESHA\aduser1

# echo ganesha > /myfs/export1/file1 # chown 15001:10001 /myfs/export1/file1 # ls -ltr /myfs/export1 total 1 -rw-r--r-- 1 GANESHA\aduser1 GANESHA\ganeshagr1 8 Jul 28 20:49 file1

3. Client mounts nfs export (/myfs/export1) using nfsv4 mount option.

# client nfsv4 mount done

192.168.122.199:/myfs/export1 /mnt/4s nfs4 rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.122.246,local_lock=none,addr=192.168.122.199 0 0

Example 1: NFSv4 ID Map Domain mismatch

1. NFSv4 ID map Domain setting on server and client are different e.g. test.com on server whereas dummy.com on client.

# server # grep ^Domain /etc/idmapd.conf Domain = test.com # client $ grep ^Domain /etc/idmapd.conf Domain = dummy.com

2. Verify on client the ownership of file file1 (created in Test Setup step 2 above).
NFS Client sees the file ownership as “nobody” – this is due to mismatched NFSv4 ID Map Domain on server and client.

# client sees nobody ownership
$ su GANESHA\aduser1 -c "ls -ltr /mnt/4s" total 1 -rw-r--r-- 1 nobody nobody 8 Jul 28 20:49 file1

3. Another example where you encounter this issue is upon creating file/dir from client e.g. client creates a file “client.file1” using user GANESHA\aduser1 and tries to get the ownership of this file, it again gets “nobody”.

# client creates file with user GANESHA\aduser1 and still sees "nobody" ownership for that file

$ su GANESHA\aduser1 -c "echo ganesha > /mnt/4s/client.file1"
$ su GANESHA\aduser1 -c "ls -ltr /mnt/4s" total 1 -rw-r--r-- 1 nobody nobody 8 Jul 28 20:49 file1 -rw-r--r-- 1 nobody nobody 8 Jul 28 20:50 client.file1

Example 2: NFSv4 ID Map Domain MATCH

1. To solve the issue observed in Example 1 above, we’ll set NFSv4 ID Mapping domain on NFS client to the same value as that of server. For this, we’ll edit /etc/idmapd.conf file and change Domain attribute to “test.com”.

# client: change NFSv4 ID Mapping domain and match it with NFS server

# client
$ vim /etc/idmapd.conf $ grep ^Domain /etc/idmapd.conf Domain = test.com

2. Restart nfs-idmap service on client.

$ nfsidmap -c; service nfs-idmap restart Redirecting to /bin/systemctl restart nfs-idmap.service

3. On NFS Client, Check the file ownership again with new idmap settings.
We can see files show the correct ownership (GANESHA\aduser1).

# client shows proper ownership under NFSv4 AUTH_SYS mount
# after matching NFSv4 ID Mapping Domain on client and server

$ su GANESHA\aduser1 -c "ls -ltr /mnt/4s" total 1 -rw-r--r-- 1 GANESHA\aduser1 GANESHA\ganeshagr1 8 Jul 28 20:49 file1 -rw-r--r-- 1 GANESHA\aduser1 GANESHA\ganeshagr1 8 Jul 28 20:50 client.file1

4. On NFS Client, Create a new file and then check it is reflected with proper ownership (i.e. owner should be set to GANESHA\aduser1 and not to nobody).

# client creates new file and gets proper ownership to that file $ su GANESHA\aduser1 -c "echo ganesha > /mnt/4s/client.file2" $ su GANESHA\aduser1 -c "ls -ltr /mnt/4s" total 2 -rw-r--r-- 1 GANESHA\aduser1 GANESHA\ganeshagr1 8 Jul 28 20:49 file1 -rw-r--r-- 1 GANESHA\aduser1 GANESHA\ganeshagr1 8 Jul 28 20:50 client.file1 -rw-r--r-- 1 GANESHA\aduser1 GANESHA\ganeshagr1 8 Jul 28 20:52 client.file2

How to configure RHEL host as LDAP client using nslcd

RHEL machine can be configured as LDAP client using nslcd.
Summarized below are the steps to configure RHEL7.6 machine as LDAP client.

1. Check whether nslcd service is already installed on your system.

$ service nslcd status Redirecting to /bin/systemctl status nslcd.service Unit nslcd.service could not be found.

2. If nslcd service is not installed, then install it using following packages.

$ yum install pam_ldap nscd nss-pam-ldapd -y ... Installing: nss-pam-ldapd ... Installed: nss-pam-ldapd.x86_64 0:0.8.13-16.el7 Complete!

$ service nslcd status Redirecting to /bin/systemctl status nslcd.service ● nslcd.service - Naming services LDAP client daemon. Loaded: loaded (/usr/lib/systemd/system/nslcd.service; disabled; vendor preset: disabled) Active: inactive (dead) Docs: man:nslcd(8) man:nslcd.conf(5)

3. Update nslcd configuration file (/etc/nslcd.conf) with your LDAP server details.
Need to update following attributes:
uri – enter your LDAP server IP/hostname here
base – enter search base for your LDAP server
binddn – user to connect with LDAP server
bindpw – credentials for the binddn user

$ vim /etc/nslcd.conf
$ egrep "^(uri|base|binddn|bindpw)" /etc/nslcd.conf uri ldap://192.168.122.250 base dc=example,dc=com binddn cn=manager,dc=example,dc=com bindpw Passw0rd

4. Update naming service configuration file (/etc/nsswitch.conf) so that RHEL machine starts looking up user information from nslcd. We need to add “ldap” entry if you want to fetch information from nslcd.

$ vim /etc/nsswitch.conf $ grep ldap /etc/nsswitch.conf passwd: files ldap shadow: files ldap group: files ldap

5. Start nslcd service.

$ service nslcd start Redirecting to /bin/systemctl start nslcd.service

6. Resolve LDAP users.

$ id developer1 uid=41001(puneldapuser1) gid=41001(puneldapgroup1) groups=41001(puneldapgroup1) $ id tester1 uid=61001(tester1) gid=51000(testers) groups=51000(testers) $ id dummyuser1 uid=2002(dummyuser1) gid=2002(dummygroup1) groups=2002(dummygroup1),31000(developers)

How to configure RHEL host as Active Directory Client using SSSD

You can configure RHEL machine as a client of Active Directory server using SSSD and AD provider. I’ve summarized the steps which worked on my test setup.

Test Setup:

>> DNS server – 192.168.122.151
>> AD server – 192.168.122.152 (win12servervm1.RAMA.HUMAN.NET)
>> AD domain – RAMA.HUMAN.NET
>> RHEL client name – robothost

Steps to configure RHEL machine as AD Client using SSSD:

1. Ensure your test set up is ready to configure AD client.

$ cat /etc/resolv.conf
nameserver 192.168.122.151

$ nslookup 192.168.122.152 Server: 192.168.122.151 Address: 192.168.122.151#53 152.122.168.192.in-addr.arpa name = win12servervm1.RAMA.HUMAN.NET.

$ dig -t SRV _ldap._tcp.rama.human.net @192.168.122.152 ; <<>> DiG 9.9.4-RedHat-9.9.4-29.el7 <<>> -t SRV _ldap._tcp.rama.human.net @192.168.122.152 ... ;; ANSWER SECTION: _ldap._tcp.rama.human.net. 600 IN SRV 0 100 389 win12servervm1.rama.human.net. ;; ADDITIONAL SECTION: win12servervm1.rama.human.net. 3600 IN A 192.168.122.152

2. Install pre requisite packages.

$ yum list samba-common* Loaded plugins: product-id, search-disabled-repos, subscription-manager This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. Installed Packages samba-common.noarch 4.2.3-10.el7 @RHEL-7.2 samba-common-libs.x86_64 4.2.3-10.el7 @RHEL-7.2 samba-common-tools.x86_64 4.2.3-10.el7 @RHEL-7.2

3. Update configuration in /etc/samba/smb.conf.
Update the entries marked in bold as per your set up (workgroup, realm).

$ testparm -s Load smb config files from /etc/samba/smb.conf rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384) Processing section "[homes]" Processing section "[printers]" Loaded services file OK. Server role: ROLE_DOMAIN_MEMBER

# Global parameters [global] workgroup = RAMA realm = RAMA.HUMAN.NET server string = Samba Server Version %v security = ADS kerberos method = secrets and keytab log file = /var/log/samba/log.%m max log size = 50 idmap config * : backend = tdb cups options = raw [homes] comment = Home Directories read only = No browseable = No [printers] comment = All Printers path = /var/spool/samba printable = Yes print ok = Yes browseable = No $ stat /etc/krb5.keytab stat: cannot stat ‘/etc/krb5.keytab’: No such file or directory

4. Join to AD domain. Here I used Administrator user (not mandatory).
You just need user account which can join a computer account to AD server.

$ net ads join -U Administrator%Passw0rd Using short domain name -- RAMA Joined 'ROBOTHOST' to dns domain 'RAMA.HUMAN.NET' No DNS domain configured for robothost. Unable to perform DNS Update. DNS update failed: NT_STATUS_INVALID_PARAMETER $ echo $? 0

5. Verify keytab entries are updated (klist -e).

$ ls /etc/krb5.keytab /etc/krb5.keytab $ klist -kte Keytab name: FILE:/etc/krb5.keytab KVNO Timestamp Principal 2 07/28/2019 16:24:38 host/robothost.rama.human.net@RAMA.HUMAN.NET (des-cbc-crc) 2 07/28/2019 16:24:38 host/robothost.rama.human.net@RAMA.HUMAN.NET (des-cbc-md5) 2 07/28/2019 16:24:38 host/robothost.rama.human.net@RAMA.HUMAN.NET (aes128-cts-hmac-sha1-96)
2 07/28/2019 16:24:38 host/robothost.rama.human.net@RAMA.HUMAN.NET (aes256-cts-hmac-sha1-96) 2 07/28/2019 16:24:38 host/robothost.rama.human.net@RAMA.HUMAN.NET (arcfour-hmac) 2 07/28/2019 16:24:38 host/robothost@RAMA.HUMAN.NET (des-cbc-crc) 2 07/28/2019 16:24:38 host/robothost@RAMA.HUMAN.NET (des-cbc-md5) 2 07/28/2019 16:24:38 host/robothost@RAMA.HUMAN.NET (aes128-cts-hmac-sha1-96) 2 07/28/2019 16:24:38 host/robothost@RAMA.HUMAN.NET (aes256-cts-hmac-sha1-96) 2 07/28/2019 16:24:38 host/robothost@RAMA.HUMAN.NET (arcfour-hmac) 2 07/28/2019 16:24:38 ROBOTHOST$@RAMA.HUMAN.NET (des-cbc-crc) 2 07/28/2019 16:24:38 ROBOTHOST$@RAMA.HUMAN.NET (des-cbc-md5) 2 07/28/2019 16:24:38 ROBOTHOST$@RAMA.HUMAN.NET (aes128-cts-hmac-sha1-96) 2 07/28/2019 16:24:38 ROBOTHOST$@RAMA.HUMAN.NET (aes256-cts-hmac-sha1-96) 2 07/28/2019 16:24:38 ROBOTHOST$@RAMA.HUMAN.NET (arcfour-hmac)

6. Update SSSD configuration file (/etc/sssd/sssd.conf).

$ cat /etc/sssd/sssd.conf [sssd] config_file_version = 2 reconnection_retries = 3 services = nss, pam domains = RAMA.HUMAN.NET [nss] filter_groups = root filter_users = root reconnection_retries = 3 [pam] reconnection_retries = 3 [domain/RAMA.HUMAN.NET] id_provider = ad auth_provider = ad access_provider = ad chpass_provider = ad ldap_id_mapping = False ad_domain = RAMA.HUMAN.NET ad_server = win12servervm1.rama.human.net override_homedir = /home/%u fallback_homedir = /home/%u

7. Start sssd service.

$ service sssd restart Redirecting to /bin/systemctl restart sssd.service

8. Add “sss” entry in name services configuration file (/etc/nsswitch.conf).

$ egrep "^(passwd|group)" /etc/nsswitch.conf passwd: files sss group: files sss

9. Resolve AD users.

$ id RAMA\\ramausr1 uid=21001(RAMA\ramausr1) gid=20001(RAMA\ramagr1) groups=20001(RAMA\ramagr1),16777217(BUILTIN\users) $ id RAMA\\user20k uid=20000(RAMA\user20k) gid=20000(RAMA\group20k) groups=20000(RAMA\group20k),16777217(BUILTIN\users)

In short, we:
– installed packages (samba-common*)
– updated configuration files (/etc/samba/smb.conf, /etc/sssd/sssd.conf)
– restarted sssd service
– attempted resolving AD users.

“These are my personal views; better ways to achieve above configuration may exist.”

mmadquery primer (apparatus to query Active Directory from IBM Spectrum Scale)

If you intend to verify configuration on AD server or run basic checks against AD server from IBM Spectrum Scale node, then mmadquery is handy utility.

mmadquery is also helpful when IBM Spectrum Scale is configured with AD RFC2307 file authentication and you want to ensure your AD users are populated with the required attributes (uidNumber, primaryGroupID, gidNumber) properly. If these attributes have out of range values, then you hit user resolution issue leading to access failures from file protocol services e.g. NFS, SMB. With mmadquery output you’ll come to know about such out of range users – then you can take corrective action of updating valid values for attributes to resolve protocol access issues.

Following is the quick summary of what mmadquery utility helps achieve :

find out domain controllers served by a domain
find out trust relationships.
list the users and groups defined on AD server, their uids and gids
get the group memberships for a user
find out total number of users per group
list the id range configured on local node
report warning about the users falling outside locally defined range
and so on…

Lets run through few examples to demonstrate the capability of mmadquery utility.
I’ve picked up following AD server IPs and domains to put across the point:

AD server ip – 192.168.122.193, domain – ADDOMAIN.COM
AD server ip – 192.168.122.151, domain – GANESHA.DEV.NET
AD server ip – 192.168.122.152, domain – RAMA.HUMAN.NET
Please note that we need an AD user having enough permissions to query user / group information from Active directory (e.g. examples below use AD user aduser1). Not necessarily an administrator user required here.
AD server ip is passed to –servers option, domain to –domain option and AD query user (e.g. aduser1) to –user option of mmadquery.

1. find out domain controllers served by a domain.

$ mmadquery list dc --server 192.168.122.193 --domain ADDOMAIN.COM --user aduser1 -L Password:
DC from server 192.168.122.193 (domain ADDOMAIN.COM) DC Hostname Operating System
————————————————————————————————-
WIN8-12UP WIN8-12Up.addomain.com Windows Server 2012 R2 Standard WIN2K16 win2k16.addomain.com Windows Server 2016 Datacenter

$ mmadquery list dc --server 192.168.122.151 --domain GANESHA.DEV.NET --user aduser1 --pwd-file /tmp/pwdfile -L DC from server 192.168.122.151 (domain GANESHA.DEV.NET) DC Hostname Operating System WIN16SERVERVM1 win16servervm1.GANESHA.DEV.NET Windows Server 2016 Standard

2. Find out the trusts set up on AD server.

$ mmadquery list trusts --server 192.168.122.151 --domain GANESHA.DEV.NET --user aduser1 --pwd-file /tmp/pwdfile TRUSTS from server 192.168.122.151 (domain GANESHA.DEV.NET) Domain Trust Type RAMA.HUMAN.NET Forest Transitive bi-directional

3. Find out what is the id range configured on local node against an AD server.

$ mmadquery list idrange --server 192.168.122.151 --domain GANESHA.DEV.NET --user aduser1 --pwd-file /tmp/pwdfile
IDRANGE from server 192.168.122.151 (domain GANESHA.DEV.NET) Domain IDRange GANESHA.DEV.NET 9000-30000

4. Find out the id range configured on local node for trusted domains. Also print the output in parsable format with the help of mmadquery -Y option. (Here assumption is that user aduser1 is present on all trusted domains.)

$ mmadquery list idrange --server 192.168.122.151 --domain GANESHA.DEV.NET --user aduser1 --pwd-file /tmp/pwdfile --traverse-domains -Y mmadquery:idrange:HEADER:version:reserved:reserved:domain:domain:idRange: mmadquery:idrange:0:1:::Default:GANESHA.DEV.NET:9000-30000: mmadquery:idrange:0:1:::Default:RAMA.HUMAN.NET:40000-50000:

5. Find out the groups defined on AD server

$ mmadquery list groups --server 192.168.122.151 --domain GANESHA.DEV.NET --user aduser1 --pwd-file /tmp/pwdfile -Y mmadquery:groups:HEADER:version:reserved:reserved:group: mmadquery:groups:0:1:::Domain Computers: mmadquery:groups:0:1:::Domain Controllers: ... mmadquery:groups:0:1:::ganeshaGr1: mmadquery:groups:0:1:::group20k: mmadquery:groups:0:1:::ganeshagr2:

6. Find out the groups with their GIDNumber.

$ mmadquery list gids --server 192.168.122.151 --domain GANESHA.DEV.NET --user aduser1 --pwd-file /tmp/pwdfile -Y mmadquery:gids:HEADER:version:reserved:reserved:group:sid:gidNumber: mmadquery:gids:0:1:::Domain Computers:S-1-5-21-3530651294-1835310728-4022169620-515:: mmadquery:gids:0:1:::Domain Controllers:S-1-5-21-3530651294-1835310728-4022169620-516:: ... mmadquery:gids:0:1:::ganeshaGr1:S-1-5-21-3530651294-1835310728-4022169620-1103:10001: mmadquery:gids:0:1:::group20k:S-1-5-21-3530651294-1835310728-4022169620-1109:20000: mmadquery:gids:0:1:::ganeshagr2:S-1-5-21-3530651294-1835310728-4022169620-1120:10002:

7. Find out the users defined on AD server.

$ mmadquery list user --server 192.168.122.151 --domain GANESHA.DEV.NET --user aduser1 --pwd-file /tmp/pwdfile -Y mmadquery:user:HEADER:version:reserved:reserved:user: mmadquery:user:0:1:::Administrator: mmadquery:user:0:1:::Guest: mmadquery:user:0:1:::DefaultAccount: mmadquery:user:0:1:::krbtgt: mmadquery:user:0:1:::ganeshaUsr1: mmadquery:user:0:1:::RAMA$: mmadquery:user:0:1:::user20k: mmadquery:user:0:1:::testuser1: mmadquery:user:0:1:::ganeshausr2: mmadquery:user:0:1:::testuser2: mmadquery:user:0:1:::aduser1: mmadquery:user:0:1:::aduser2:

8. Find out the users with their UIDNumber.

$ mmadquery list uids --server 192.168.122.151 --domain GANESHA.DEV.NET --user aduser1 --pwd-file /tmp/pwdfile -Y mmadquery:uids:HEADER:version:reserved:reserved:user:sid:uid:uidNumber: mmadquery:uids:0:1:::Administrator:S-1-5-21-3530651294-1835310728-4022169620-500::: mmadquery:uids:0:1:::Guest:S-1-5-21-3530651294-1835310728-4022169620-501::: mmadquery:uids:0:1:::DefaultAccount:S-1-5-21-3530651294-1835310728-4022169620-503::: mmadquery:uids:0:1:::krbtgt:S-1-5-21-3530651294-1835310728-4022169620-502::: mmadquery:uids:0:1:::ganeshaUsr1:S-1-5-21-3530651294-1835310728-4022169620-1104::11001: mmadquery:uids:0:1:::RAMA$:S-1-5-21-3530651294-1835310728-4022169620-1108::: mmadquery:uids:0:1:::user20k:S-1-5-21-3530651294-1835310728-4022169620-1110::20000: mmadquery:uids:0:1:::testuser1:S-1-5-21-3530651294-1835310728-4022169620-1115::13001: mmadquery:uids:0:1:::ganeshausr2:S-1-5-21-3530651294-1835310728-4022169620-1116::11002: mmadquery:uids:0:1:::testuser2:S-1-5-21-3530651294-1835310728-4022169620-1117::13002: mmadquery:uids:0:1:::aduser1:S-1-5-21-3530651294-1835310728-4022169620-1118::15001: mmadquery:uids:0:1:::aduser2:S-1-5-21-3530651294-1835310728-4022169620-1119::15002:

9. Find out group list of each user (i.e. user and groups to which that user belongs to).

$ mmadquery list user --server 192.168.122.151 --domain GANESHA.DEV.NET --user aduser1 --pwd-file /tmp/pwdfile -L -Y mmadquery:user_long:HEADER:version:reserved:reserved:user:groups: mmadquery:user_long:0:1:::Administrator:Group Policy Creator Owners,Domain Admins,Enterprise Admins,Schema Admins,Administrators: mmadquery:user_long:0:1:::Guest:Guests: mmadquery:user_long:0:1:::DefaultAccount:System Managed Accounts Group: mmadquery:user_long:0:1:::krbtgt:Denied RODC Password Replication Group: mmadquery:user_long:0:1:::ganeshaUsr1:ganeshagr2,group20k,Domain Users: mmadquery:user_long:0:1:::RAMA$:: mmadquery:user_long:0:1:::user20k:Domain Users: mmadquery:user_long:0:1:::testuser1:Domain Users: mmadquery:user_long:0:1:::ganeshausr2:Domain Users: mmadquery:user_long:0:1:::testuser2:: mmadquery:user_long:0:1:::aduser1:group20k,Domain Users: mmadquery:user_long:0:1:::aduser2:Domain Users:

10. Find out the users with their uid and gid details.
If values for UIDNumber, GIDNumber and Primary Group GID are out of range, then you face access issues via NFS/SMB. This topic needs an explanation and is covered under another blog.

$ mmadquery list uids --server 192.168.122.151 --domain GANESHA.DEV.NET --user aduser1 --pwd-file /tmp/pwdfile -L -Y mmadquery:uids_long:HEADER:version:reserved:reserved:user:sid:uid:uidNumber:gidNumber:primaryGroupId:primaryGroupGid: mmadquery:uids_long:0:1:::Administrator:S-1-5-21-3530651294-1835310728-4022169620-500::::513:: mmadquery:uids_long:0:1:::Guest:S-1-5-21-3530651294-1835310728-4022169620-501::::514:: mmadquery:uids_long:0:1:::DefaultAccount:S-1-5-21-3530651294-1835310728-4022169620-503::::513:: mmadquery:uids_long:0:1:::krbtgt:S-1-5-21-3530651294-1835310728-4022169620-502::::513:: mmadquery:uids_long:0:1:::ganeshaUsr1:S-1-5-21-3530651294-1835310728-4022169620-1104::11001:10001:1103:10001: mmadquery:uids_long:0:1:::RAMA$:S-1-5-21-3530651294-1835310728-4022169620-1108::::513:: mmadquery:uids_long:0:1:::user20k:S-1-5-21-3530651294-1835310728-4022169620-1110::20000:20000:1109:20000: mmadquery:uids_long:0:1:::testuser1:S-1-5-21-3530651294-1835310728-4022169620-1115::13001::1103:10001: mmadquery:uids_long:0:1:::ganeshausr2:S-1-5-21-3530651294-1835310728-4022169620-1116::11002:20000:1103:10001: mmadquery:uids_long:0:1:::testuser2:S-1-5-21-3530651294-1835310728-4022169620-1117::13002:20000:513:: mmadquery:uids_long:0:1:::aduser1:S-1-5-21-3530651294-1835310728-4022169620-1118::15001::1103:10001: mmadquery:uids_long:0:1:::aduser2:S-1-5-21-3530651294-1835310728-4022169620-1119::15002::1103:10001:

11. Find out the total number of users per group

$ mmadquery stats user --server 192.168.122.151 --domain GANESHA.DEV.NET --user aduser1 --pwd-file /tmp/pwdfile -Y mmadquery:user_stats:HEADER:version:reserved:reserved:group:count: mmadquery:user_stats:0:1:::TOTAL:12: mmadquery:user_stats:0:1:::Group Policy Creator Owners:1: mmadquery:user_stats:0:1:::Domain Admins:1: mmadquery:user_stats:0:1:::Enterprise Admins:1: mmadquery:user_stats:0:1:::Schema Admins:1: mmadquery:user_stats:0:1:::Administrators:1: mmadquery:user_stats:0:1:::Guests:1: mmadquery:user_stats:0:1:::System Managed Accounts Group:1: mmadquery:user_stats:0:1:::Denied RODC Password Replication Group:1: mmadquery:user_stats:0:1:::ganeshagr2:1: mmadquery:user_stats:0:1:::group20k:2: mmadquery:user_stats:0:1:::Domain Users:6:

12. Find out the total number of mapped/un-mapped/total users on local node.

$ mmadquery stats uids --server 192.168.122.151 --domain GANESHA.DEV.NET --user aduser1 --pwd-file /tmp/pwdfile --traverse-domains -Y mmadquery:uids_stats:HEADER:version:reserved:reserved:domain:group:count: mmadquery:uids_stats:0:1:::GANESHA.DEV.NET:TOTAL:12: mmadquery:uids_stats:0:1:::GANESHA.DEV.NET:MAPPED:7: mmadquery:uids_stats:0:1:::GANESHA.DEV.NET:UN-MAPPED:5: mmadquery:uids_stats:0:1:::RAMA.HUMAN.NET:TOTAL:6: mmadquery:uids_stats:0:1:::RAMA.HUMAN.NET:MAPPED:2: mmadquery:uids_stats:0:1:::RAMA.HUMAN.NET:UN-MAPPED:4:

13. Find out uid of AD user falls within locally defined id mapping range or not.

If user’s uid is outside locally defined id range, then mmadquery reports WARNING message e.g. “WARNING: UID of user ‘ganeshaUsr1’ outside id mapping range 10000000-299999999 for domain ‘GANESHA.DEV.NET’.”

If user’s uid is within locally defined id range, then mmadquery reports “OK, no problems found.”.

$ mmadquery check uids --server 192.168.122.152 --domain RAMA.HUMAN.NET --user Administrator --pwd-file /tmp/pwdfile -Y mmadquery:uids:HEADER:version:reserved:reserved:user:sid:uid:uidNumber: mmadquery:uids:0:1:::Administrator:S-1-5-21-2161741167-2001596563-2425738677-500::: mmadquery:uids:0:1:::Guest:S-1-5-21-2161741167-2001596563-2425738677-501::: mmadquery:uids:0:1:::krbtgt:S-1-5-21-2161741167-2001596563-2425738677-502::: mmadquery:uids:0:1:::ramaUsr1:S-1-5-21-2161741167-2001596563-2425738677-1106::21001: mmadquery:uids:0:1:::GANESHA$:S-1-5-21-2161741167-2001596563-2425738677-1107::: mmadquery:uids:0:1:::user20k:S-1-5-21-2161741167-2001596563-2425738677-1109::20000: mmadquery:check_results:HEADER:version:reserved:reserved:message_enc: mmadquery:check_results:0:1:::OK, no problems found.:

An example of user’s uid/gid falls outside range, such users reported with warning as below:

$ mmadquery check uids --server 192.168.122.151 --domain GANESHA.DEV.NET --user Administrator --pwd-file /tmp/pwdfile UIDS from server 192.168.122.151 (domain GANESHA.DEV.NET) User SID UID UIDNumber ------------------------------------------------------------------------- Administrator S-1-5-21-3530651294-1835310728-4022169620-500 - - Guest S-1-5-21-3530651294-1835310728-4022169620-501 - - DefaultAccount S-1-5-21-3530651294-1835310728-4022169620-503 - - krbtgt S-1-5-21-3530651294-1835310728-4022169620-502 - - ganeshaUsr1 S-1-5-21-3530651294-1835310728-4022169620-1104 - 11001 RAMA$ S-1-5-21-3530651294-1835310728-4022169620-1108 - - user20k S-1-5-21-3530651294-1835310728-4022169620-1110 - 20000 testuser1 S-1-5-21-3530651294-1835310728-4022169620-1115 - 13001 ganeshausr2 S-1-5-21-3530651294-1835310728-4022169620-1116 - 11002 testuser2 S-1-5-21-3530651294-1835310728-4022169620-1117 - 13002 aduser1 S-1-5-21-3530651294-1835310728-4022169620-1118 - 15001 aduser2 S-1-5-21-3530651294-1835310728-4022169620-1119 - 15002 WARNING: UID of user 'ganeshaUsr1' outside id mapping range 10000000-299999999 for domain 'GANESHA.DEV.NET'. WARNING: UID of user 'user20k' outside id mapping range 10000000-299999999 for domain 'GANESHA.DEV.NET'. WARNING: UID of user 'testuser1' outside id mapping range 10000000-299999999 for domain 'GANESHA.DEV.NET'. WARNING: UID of user 'ganeshausr2' outside id mapping range 10000000-299999999 for domain 'GANESHA.DEV.NET'. WARNING: UID of user 'testuser2' outside id mapping range 10000000-299999999 for domain 'GANESHA.DEV.NET'. WARNING: UID of user 'aduser1' outside id mapping range 10000000-299999999 for domain 'GANESHA.DEV.NET'. WARNING: UID of user 'aduser2' outside id mapping range 10000000-299999999 for domain 'GANESHA.DEV.NET'.

Understanding and Solving NT_STATUS_INVALID_SID issue for SMB access with Spectrum Scale

If you have configured IBM Spectrum Scale with LDAP authentication and then encounter NT_STATUS_INVALID_SID error while accessing IBM Spectrum Scale samba shares, then ensure your cluster netbios sambaSID is part of LDAP user sambaSID. An example to illustrate the failure and proposed solution.

1. Cluster configured with LDAP File Authentication.

$ mmuserauth service create --data-access-method file --type ldap --servers ldapkdc --base-dn dc=example,dc=com --user-name cn=manager,dc=example,dc=com --pwd-file ldap.pwdfile --netbios-name testcluster File authentication configuration completed successfully.

2. sambaSID for netbios name is not part of sambaSID for LDAP user.
Cluster netbios sambaSID: S-1-5-21-2824773955-2879828300-3619840358
LDAP user ldapuser3 sambaSID: S-1-5-21-2648479386-31017536-2378919642-3004

$ ldapsearch -x -D "cn=Manager,dc=example,dc=com" -w Passw0rd -b "dc=example,dc=com" sambaDomainName=TESTCLUSTER sambaSID ... # TESTCLUSTER, example.com dn: sambaDomainName=TESTCLUSTER,dc=example,dc=com sambaSID: S-1-5-21-2824773955-2879828300-3619840358 $ ldapsearch -x -D "cn=Manager,dc=example,dc=com" -w Passw0rd -b "dc=example,dc=com" cn=ldapuser3 sambaSID ... # ldapuser3, People, example.com dn: uid=ldapuser3,ou=People,dc=example,dc=com sambaSID: S-1-5-21-2648479386-31017536-2378919642-3004

3. When I tried to access samba shares using LDAP user ldapuser3, NT_STATUS_INVALID_SID error was thrown.

$ smbclient -L testcluster -U ldapuser3%Passw0rd -m SMB3 session setup failed: NT_STATUS_INVALID_SID

4. Corrected the sambaSID for netbios name.

$ cat changesid.ldif dn: sambaDomainName=TESTCLUSTER,dc=example,dc=com changetype: modify replace: sambaSID sambaSID: S-1-5-21-2648479386-31017536-2378919642 $ ldapmodify -x -D cn=manager,dc=example,dc=com -w Passw0rd -f changesid.ldif modifying entry "sambaDomainName=TESTCLUSTER,dc=example,dc=com"

5. Verify netbios name sambaSID (S-1-5-21-2648479386-31017536-2378919642) is part of LDAP user ldapuser3 sambaSID (S-1-5-21-2648479386-31017536-2378919642-3004).

$ ldapsearch -x -D "cn=Manager,dc=example,dc=com" -w Passw0rd -b "dc=example,dc=com" sambaDomainName=TESTCLUSTER sambaSID # TESTCLUSTER, example.com dn: sambaDomainName=TESTCLUSTER,dc=example,dc=com sambaSID: S-1-5-21-2648479386-31017536-2378919642 $ ldapsearch -x -D "cn=Manager,dc=example,dc=com" -w Passw0rd -b "dc=example,dc=com" cn=ldapuser3 sambaSID # ldapuser3, People, example.com dn: uid=ldapuser3,ou=People,dc=example,dc=com sambaSID: S-1-5-21-2648479386-31017536-2378919642-3004

6. Try to access Samba share using the same LDAP user e.g. ldapuser3 here. It succeeded.

$ smbclient -L testcluster -U ldapuser3%Passw0rd -m SMB2 Domain=[TESTCLUSTER] OS=[] Server=[] Sharename Type Comment --------- ---- ------- IPC$ IPC IPC Service (IBM NAS) smbexp1 Disk smbexp3 Disk smbexp_fset1 Disk Domain=[TESTCLUSTER] OS=[] Server=[] Server Comment --------- ---- ------- Workgroup Master --------- ---- -------

Understanding and Solving WBC_ERR_DOMAIN_NOT_FOUND error with Spectrum Scale

Problem statement

If IBM Spectrum Scale is configured with AD RFC2307 file authentication and you face either of the following issues:

1) NFS protocol access issue with some AD user (e.g. NFS permission denied)
2) SMB protocol access issue with AD user (NT_STATUS_LOGON_FAILURE)
3) failure in resolving some AD user on IBM Spectrum Scale protocol node

then one of the reason could be WBC_ERR_DOMAIN_NOT_FOUND / NT_STATUS_NONE_MAPPED error under the hood.

This blog explains the possible solution for this error. It leverages capability of mmadquery utility in IBM Spectrum Scale to determine the failure cause.

Failure Examples

1. file protocol access issue (e.g. NFS permission denied) with some AD user

# WBC_ERR_DOMAIN_NOT_FOUND in NFS server log for permission denied issue
2019-03-26 15:00:09 : epoch 0003004b : rh73n2 : gpfs.ganesha.nfsd-15043[work-101] nfs_req_creds :ID MAPPER :WARN :Could not map principal user1@XYZ.COM to uid 2019-03-26 15:00:09 : epoch 0003004b : rh73n2 : gpfs.ganesha.nfsd-15043[work-102] principal2uid :ID MAPPER :CRIT :wbcSidToUid for uid returned WBC_ERR_DOMAIN_NOT_FOUND

2. SMB protocol access issue with AD user (NT_STATUS_LOGON_FAILURE)

# NT_STATUS_NONE_MAPPED error reported in winbind.log for SMB access
$ smbclient //rh73n/smbexp_fset1 -m smb3 -U GANESHA\aduser2%Passw0rd -c "ls" session setup failed: NT_STATUS_LOGON_FAILURE

3. Failure in resolving AD user

# WBC_ERR_DOMAIN_NOT_FOUND while resolving AD user on IBM Spectrum Scale
$ /usr/lpp/mmfs/bin/wbinfo -i GANESHA\\aduser1 failed to call wbcGetpwnam: WBC_ERR_DOMAIN_NOT_FOUND Could not get info for user GANESHA\aduser1

Solution for above failures

Following will help to solve above failures. Ensure:
– uidNumber and primaryGroupID set for that user
– uidNumber and primaryGroupID values must fall in –unixmap-domains range

An example to explicate the problem and suggested solution.

1. IBM Spectrum Scale configured with AD RFC2307 file authentication with “GANESHA” domain and uid/gid range “9000-30000”.

$ mmuserauth service create --data-access-method file --type ad --servers 192.168.122.151 --user-name Administrator --pwd-file adprivate.pwdfile --netbios-name rh73n --idmap-role master --unixmap-domains "GANESHA(9000-30000)" File authentication configuration completed successfully.

2. Test Setup of two users from AD:
I’ve defined two users in AD as below:

aduser1 – uidNumber and primaryGroupID not set
aduser2 – primaryGroupID not set

(For reference, see image below for user aduser1 uidNumber and primaryGroupID.
– primaryGroupID not set because “Domain Users” GID doesn’t fall within –unixmap-domains range “9000-30000” configured on the cluster
– uidNumber is not set)

3. Verification using mmadquery command for problematic users:
You can also check above settings on IBM Spectrum Scale using mmadquery command. Field “uidNumber” displays uidNumber and field “primaryGroupGid” displays primaryGroupID.

In the following output, you can validate that uidNumber and primaryGroupID not set for aduser1, primaryGroupID not set for aduser2. This will lead to the problems mentioned in “Failure Examples” section above.

$ mmadquery list uids --server 192.168.122.151 --domain GANESHA.DEV.NET --user Administrator --pwd-file /tmp/pwdfile -L -Y mmadquery:uids_long:HEADER:version:reserved:reserved:user:sid:uid:uidNumber:gidNumber:primaryGroupId:primaryGroupGid: mmadquery:uids_long:0:1:::aduser1:S-1-5-21-3530651294-1835310728-4022169620-1118::::1103:: mmadquery:uids_long:0:1:::aduser2:S-1-5-21-3530651294-1835310728-4022169620-1119::15002::1103::

4. Try to resolve AD users aduser1 and aduser2 – it fails (due to missing uidNumber and/or primaryGroupID values).

$ /usr/lpp/mmfs/bin/wbinfo -a GANESHA\\aduser1%Passw0rd plaintext password authentication succeeded challenge/response password authentication succeeded $ /usr/lpp/mmfs/bin/wbinfo -i GANESHA\\aduser1 failed to call wbcGetpwnam: WBC_ERR_DOMAIN_NOT_FOUND Could not get info for user GANESHA\aduser1 $ /usr/lpp/mmfs/bin/wbinfo -a GANESHA\\aduser2%Passw0rd plaintext password authentication succeeded challenge/response password authentication succeeded $ /usr/lpp/mmfs/bin/wbinfo -i GANESHA\\aduser2 failed to call wbcGetpwnam: WBC_ERR_DOMAIN_NOT_FOUND Could not get info for user GANESHA\aduser2

5. Set uidNumber and primaryGroupID for the users in Active Directory:
On AD server, Set uidNumber and primaryGroupID for aduser1.
Set primaryGroupID for aduser2. Make sure uidNumber and primaryGroupID values fall within –unixmap-domains range configured on cluster.

– uidNumber can be set through “Attribute Editor” -> uidNumber attribute.
– primaryGroupID can be set through “Members Of” -> Set primary group to Group that falls within –unixmap-domains range

(For reference, modified values for aduser1 are shown in following image
uidNumber=15001 and primaryGroupID=10001. Both these values are within –unixmap-domains range which is “9000-30000”.)

6. Verification using mmadquery command.
You can verify the modified uidNumber and primaryGroupID on IBM Spectrum scale using mmadquery command.

$ mmadquery list uids --server 192.168.122.151 --domain GANESHA.DEV.NET --user Administrator --pwd-file /tmp/pwdfile -L -Y mmadquery:uids_long:HEADER:version:reserved:reserved:user:sid:uid:uidNumber:gidNumber:primaryGroupId:primaryGroupGid:
mmadquery:uids_long:0:1:::aduser1:S-1-5-21-3530651294-1835310728-4022169620-1118::15001::1103:10001: mmadquery:uids_long:0:1:::aduser2:S-1-5-21-3530651294-1835310728-4022169620-1119::15002::1103:10001:

7. Resolving AD users succeed.
Try to resolve AD users again. They do succeed.

$ /usr/lpp/mmfs/bin/wbinfo -a GANESHA\\aduser1%Passw0rd plaintext password authentication succeeded challenge/response password authentication succeeded $ /usr/lpp/mmfs/bin/wbinfo -i GANESHA\\aduser1 GANESHA\aduser1:*:15001:10001:aduser1:/home/GANESHA/aduser1:/bin/false $ /usr/lpp/mmfs/bin/wbinfo -a GANESHA\\aduser2%Passw0rd plaintext password authentication succeeded challenge/response password authentication succeeded $ /usr/lpp/mmfs/bin/wbinfo -i GANESHA\\aduser2 GANESHA\aduser2:*:15002:10001:aduser2:/home/GANESHA/aduser2:/bin/false

8. SMB access succeeds:
Try to perform SMB access with AD user aduser2 (for example 3 failure). It succeeds this time because primaryGroupID for aduser2 is now set.

$ smbclient //rh73n/smbexp_fset1 -m smb3 -U GANESHA\aduser2%Passw0rd -c "ls" Domain=[GANESHA] OS=[] Server=[] . D 0 Mon Jul 15 23:45:29 2019 .. D 0 Sat Jul 20 00:02:57 2019 file1 A 8 Mon Jul 15 23:45:29 2019

In above example, IBM Spectrum Scale was configured to fetch primary group as Windows primary group of a user on the Active Directory. It is identified by primaryGroupID attribute on Active Directory.

You can also configure IBM Spectrum Scale to fetch primary group as set in “UNIX attributes” of a user on the Active Directory e.g. –unixmap-domains “GANESHA(20000-50000:unix)”. In such case, you need to ensure gidNumber is set for AD user to avoid above failures (mmadquery displays it as gidNumber field).

“These are my personal views and do not necessarily reflect that of my employer” .

Enforce AES encrypted NFS data traffic from IBM Spectrum Scale

Introduction

IBM Spectrum Scale can be configured with Kerberos for secure data access over NFS. For details, one can refer technote “Accessing IBM Spectrum Scale over NFS Kerberos using LDAP and MIT KDC” (https://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/TD106395).

This blog explains how to ensure secure data access over NFS with strong encryption type AES-128 or AES-256 i.e. NFS server allows NFS client to communicate with AES encryption algorithm only.

We can achieve it by restricting permitted_enctypes to AES in kerberos configuration file on IBM Spectrum Scale.

permitted_enctypes option is verified against following KDCs:

MIT KDC – krb5-server-1.14.1-26 on RHEL7.3 x86_64
Active Directory KDC – Windows Server 2012 Standard
Active Directory KDC – Windows Server 2016 Standard

For brevity, sample examples are provided for MIT-KDC only (by configuring IBM Spectrum Scale with LDAP+Kerberos file authentication). It works for Active Directory KDCs too (if you configure AD+RFC2307+Kerberos file authentication on IBM Spectrum Scale).

Example 1: showcases positive scenario i.e. NFS server to NFS client communication with AES encryption.
Example 2: demonstrates negative scenario where NFS server rejects request from NFS client coming in with weak enctype (e.g. arcfour-hmac).

Examples refer :
– “rh73n” as IBM Spectrum Scale NFS server (on RHEL7.3 x86_64)
– “ldapclient” as NFS client (on RHEL7.2 x86_64)
– “ldapkdc” as MIT KDC server (on RHEL7.3 x86_64)
– AES-256 enctype as 18 numeric identifier
– arcfour-hmac enctype as 23 numeric identifier
(For enctypes to numeric identifier mapping, refer https://tools.ietf.org/html/rfc3961#section-8.)

IBM Spectrum Scale Setup for Example 1 and Example 2:

1. Configure File authentication with LDAP and MIT KDC.

$ mmuserauth service create --data-access-method file --type ldap --servers ldapkdc --user-name cn=manager,dc=example,dc=com --base-dn dc=example,dc=com --enable-kerberos --netbios-name rh73n --kerberos-server ldapkdc --kerberos-realm EXAMPLE.COM Enter LDAP User 'cn=manager,dc=example,dc=com' password: File authentication configuration completed successfully.

2. Create NFS export with krb5p security flavor.

$ mmnfs export add /ibm/gpfs0/fset1 -c "*(Access_Type=RW,squash=no_root_squash,SecType=krb5p)" mmnfs: The NFS export was created successfully

3. Restrict permitted_enctypes to AES only

On IBM Spectrum Scale protocol nodes, restrict encryption types to AES only. You can do this by adding permitted_enctypes line under “[libdefaults]” section in Kerberos configuration file /etc/krb5.conf. Set permitted_enctypes with AES encryption only (e.g. aes128-cts-hmac-sha1-96,aes256-cts-hmac-sha1-96 ).

Execute following commands to update /etc/krb5.conf on all protocol nodes.

$ mmccr fget KRB5_CONF /tmp/krb5.conf.orig $ cp /tmp/krb5.conf.orig /tmp/krb5.conf.new $ vim /tmp/krb5.conf.new # add permitted_enctypes under [libdefaults] section $ diff /tmp/krb5.conf.orig /tmp/krb5.conf.new permitted_enctypes = aes128-cts-hmac-sha1-96,aes256-cts-hmac-sha1-96 $ mmccr fput KRB5_CONF /tmp/krb5.conf.new $ mmuserauth service check --data-access-method file -N cesnodes -r $ grep permitted_enctypes /etc/krb5.conf permitted_enctypes = aes128-cts-hmac-sha1-96,aes256-cts-hmac-sha1-96

Example 1: NFS Server and NFS Client AES communication

In this example, we are trying to validate that NFS client to NFS server communication is happening with AES enctype.

Keep default values for default_tgs_enctypes and default_tkt_enctypes in /etc/krb5.conf on NFS client. It lets NFS client to request with AES enctype. This is because AES enctypes are first in the list for default_tgs_enctypes and default_tkt_enctypes. So, you don’t need to set anything explicitly for this example on NFS client side. Just try out NFS mount and access.

1. Mount NFSv3 krb5p export on NFS client.

# mount -t nfs -o vers=3,sec=krb5p rh73n:/ibm/gpfs0/fset1 /tmp/3krb5p_fset1 -vvvv mount.nfs: timeout set for Mon Jul 15 19:09:16 2019 mount.nfs: trying text-based options 'vers=3,sec=krb5p,addr=192.168.122.199' mount.nfs: prog 100003, trying vers=3, prot=6 mount.nfs: trying 192.168.122.199 prog 100003 vers 3 prot TCP port 2049 mount.nfs: prog 100005, trying vers=3, prot=17 mount.nfs: trying 192.168.122.199 prog 100005 vers 3 prot UDP port 54708

2. Access NFSv3 krb5p mountpoint. Verify NFS communication is happening over AES enctype using either of the following:
i) On NFS client, check “klist -e” output for “Etype (skey)”.
ii) On KDC, check /var/log/krb5kdc.log for “ses=xx” in TGS_REQ line e.g. etypes {rep=18 tkt=18 ses=18}.

a) Get TGT (kinit)

# su ldapuser3 -c "echo Passw0rd | kinit" Password for ldapuser3@EXAMPLE.COM:

b) Create a file under NFSv3 krb5p mountpoint. This triggers communication between NFS server and NFS client.
# su ldapuser3 -c "echo ganesha > /tmp/3krb5p_fset1/ldapuser3.file1; ls -ltr /tmp/3krb5p_fset1" total 1 -rw-r--r-- 1 ldapuser3 ldapuser3 8 Jul 15 2019 ldapuser3.file1

c) Verify communication is happening over AES enctype (Etype(skey)).
# su ldapuser3 -c "klist -e" Ticket cache: KEYRING:persistent:1002:1002 Default principal: ldapuser3@EXAMPLE.COM Valid starting Expires Service principal 07/15/2019 19:08:29 07/16/2019 19:07:37 nfs/rh73n@EXAMPLE.COM Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96 ...

# KDC log TGS_REQ line reporting ses=18 for NFS server communication
# (18 in ses=18 stands for AES-256 encryption algorithm)
Jul 15 19:08:29 ldapkdc krb5kdc1112: TGS_REQ (6 etypes {18 17 16 23 25 26}) 192.168.122.243: ISSUE: authtime 1563197897, etypes {rep=18 tkt=18 ses=18}, ldapuser3@EXAMPLE.COM for nfs/rh73n@EXAMPLE.COM

Example 2: NFS server rejects weak enctype requests from NFS client

If NFS client tries with weak enctype, then IBM Spectrum Scale NFS server rejects mount request itself. This ensures that weak enctype requests from NFS clients are not served.

1. Configure NFS client to request with weak enctypes (e.g. set default_tgs_enctypes and default_tkt_enctypes to arcfour-hmac in /etc/krb5.conf).

# grep default_.*_enctypes /etc/krb5.conf default_tgs_enctypes = arcfour-hmac default_tkt_enctypes = arcfour-hmac

2. NFS client tries NFSv3 krb5p mount – fails (as expected).
This is because KDC provides “arcfour-hmac” enctype session key to NFS client to communicate with NFS server. However, NFS server is configured to support AES only enctype, hence, NFS client’s mount request is rejected. This ensures IBM Spectrum Scale supports only AES enctype and rejects weak enctype requests.

# mount -t nfs -o vers=3,sec=krb5p rh73n:/ibm/gpfs0/fset1 /tmp/3krb5p_fset1 -vvvv mount.nfs: timeout set for Mon Jul 15 19:10:26 2019 mount.nfs: trying text-based options 'vers=3,sec=krb5p,addr=192.168.122.199' mount.nfs: prog 100003, trying vers=3, prot=6 mount.nfs: trying 192.168.122.199 prog 100003 vers 3 prot TCP port 2049 mount.nfs: prog 100005, trying vers=3, prot=17 mount.nfs: trying 192.168.122.199 prog 100005 vers 3 prot UDP port 54708 mount.nfs: mount(2): Permission denied mount.nfs: access denied by server while mounting rh73n:/ibm/gpfs0/fset1

#KDC log TGS_REQ reporting ses=23 for NFS server communication
# (23 in ses=23 stands for arcfour-hmac encryption algorithm)
Jul 15 19:09:05 ldapkdc krb5kdc1112: AS_REQ (1 etypes {23}) 192.168.122.243: ISSUE: authtime 1563197945, etypes {rep=23 tkt=18 ses=23}, nfs/ldapclient@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM Jul 15 19:09:05 ldapkdc krb5kdc1112: TGS_REQ (1 etypes {23}) 192.168.122.243: ISSUE: authtime 1563197945, etypes {rep=23 tkt=18 ses=23}, nfs/ldapclient@EXAMPLE.COM for nfs/rh73n@EXAMPLE.COM

Reference Links

1. Accessing IBM Spectrum Scale over NFS Kerberos using LDAP and MIT KDC
https://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/TD106395
2. Enctype to numeric identifier mapping
https://tools.ietf.org/html/rfc3961#section-8

“ These are my personal views and do not necessarily reflect that of my employer” .