This is a small departure from my more recent computer centric posts, but I thought it was worth posting, since most of the directions online are text only. I’ve changed belts on Singers with external motors before, but this was my first time changing an internal belt. This post should probably help anyone with a Singer 327/328, a 337/338, and a 347/348, since they’re all very similar machines. The basics should be the same for other Singers from the same era, and probably some other brands as well.

The first thing to do is to lay the machine on its side, taking it out of the cabinet or the bottom of the portable case if you have one. I like to lay mine on top of a folded towel. Look at the side with the hand wheel. On the 338, there are two large screws on the side panel. You will need to undo these. I also took off the top of my machine, but I think that was probably unnecessary.

Pretend you’re looking at a torn and tattered belt. I forgot to take before pictures, so you’re looking at my shiny new belt here. The next thing to undo is the chrome part in the center of the hand wheel (what you normally loosen before winding bobbins). You will have to first unscrew the tiny little stop screw, then unscrew the entire chrome piece until it comes out.

Now it’s time to take off the oil drip pan (if your machine has one, it’s the plastic piece that covers the bottom of the machine, generally attached with a single screw) and undo the bottom motor mount screw. On the 338, the screw for the motor mount is in a tight spot that you can only get to through the cutouts in the bottom side of the machine. The screw passes through two nylon washers, which are divided by the metal piece that actually attaches to the motor.

Here’s how I unscrewed it:

And here’s another peek, so you can see what I’m talking about. Nylon parts in old Singer sewing machines are always that white beige color.

Once it’s unscrewed, the entire motor mount will move freely up and down. You only have to nudge it a little towards the top of the machine, though, so before you get too crazy, go ahead and pull the hand wheel away from the body of the machine just enough to expose the top of the belt.

Remove the old belt (if you haven’t already) and place the new belt in the hand wheel groove. Push the wheel back into the base of the machine and loop the other end of the belt over the bolt like protrusion that sticks out of the motor. The second picture I posted above shows a correctly installed belt. You can go ahead and screw the hand wheel back on to secure it, but don’t put the side panel back on until after you reattach the bottom motor mount screw and its two nylon washers. The motor will be under tension from the belt, so this can be a little tricky. Try not to screw it in crooked. I put my left hand in through the open side panel and maneuvered the motor into position that way. Nylon parts get brittle with age and can break, so just be careful of the any white plastic pieces inside your machine.

Once the belt is securely in place, turn the hand wheel and make sure everything works. If you can’t get the chrome center part of the hand wheel to tighten correctly (the stop screw hits the washer underneath too soon before the chrome part is sufficiently secure), the in between metal washer that sat between the chrome disk and the hand wheel may have been put in the wrong way around. Flip it 180 degrees and try again. (It has two flanges that want to sit in cut outs facing into the hand wheel, but there’s nothing to show which way is correct.) If everything looks good, put the oil pan back on, put the machine back in an upright position, and try sewing.

I updated my previous post, Mac OS X 10.7 (and 10.8!), to correct some tricky errors. I had been forced to stop at good enough, but now that I’m migrating our Macs to Mountain Lion, I was able to knock out some more Kerberos bugs. Please pay careful attention to the changes to /etc/pam.d/authorization, /etc/pam.d/sudo, and /etc/pam.d/sshd. Unfortunately, the same bug that was in Lion’s implementation of SSHD appears to be present in Mountain Lion.

My inexperience with pam is mostly to blame. Here are the changes again:

For logins:

# authorization: auth account
auth       sufficient     pam_krb5.so use_first_pass default_principal
auth       optional       pam_ntlm.so use_first_pass
auth       required       pam_opendirectory.so use_first_pass nullok
account    required       pam_opendirectory.so

For LDAP enabled sudo access:

# sudo: auth account password session
auth       sufficient     pam_krb5.so try_first_pass default_principal
auth       required       pam_opendirectory.so use_first_pass
account    required       pam_permit.so
password   required       pam_deny.so
session    required       pam_permit.so

For SSH logins:

# sshd: auth account password session
auth       sufficient     pam_krb5.so try_first_pass default_principal
auth       optional       pam_ntlm.so try_first_pass
auth       optional       pam_mount.so try_first_pass
auth       required       pam_opendirectory.so try_first_pass
account    required       pam_nologin.so
account    required       pam_sacl.so sacl_service=ssh
account    required       pam_opendirectory.so
password   required       pam_opendirectory.so
session    required       pam_launchd.so
session    optional       pam_mount.so

Other applications (like su) have their own configuration files and may need similar adjustments. I was able to enable most of these for Kerberos by ensuring that the first auth line looked like this:

auth       sufficient     pam_krb5.so try_first_pass default_principal

If you find that local user accounts are getting prompted twice for a password, it’s because it tries Kerberos first and fails. To eliminate the second prompt, add the option use_first_pass to the end of the auth line for pam_opendirectory.so

It’s terrible, but it works. I needed a way to add special kernel options to certain pieces of hardware on my network. Facter (no doubt in an effort to be backwards compatible and friendly) is lacking some really spiffy dmidecode values. If you’ve never used this tool, go ahead and run it like so:

sudo dmidecode -s

This will force it to print out a list of supported strings. I needed system-product-name, specifically to test if it equals OptiPlex 990. I will use this as my example, but anything listed will work. Some of these values are already in facter, though, so double check to see if you have to do through these extra steps.

In Puppet, I use modules to organize things and keep everything tidy. Under my main modules directory, I have a module folder labeled kernel, which already contained a subfolder labeled manifests to hold my init.pp file. Now I add a folder at the same level, called lib, and a subfolder inside of that, labeled facter. Since I’ve already been using plugins, I already have “pluginsync = true” in [main] in my puppet.conf file. You should do the same. Anyhow, once I created these two additional folders, I went ahead and created the file modules/kernel/lib/facter/systemproductname.rb. Here’s what I put inside:

Facter.add("systemproductname") do
  confine :kernel => :linux
    setcode do
      Facter::Util::Resolution.exec("/usr/sbin/dmidecode -s system-product-name")
    end
end

Facter::Util::Resolution is an api function for Facter provided by PuppetLabs. Ruby kindly returns its output without needing to set it to any intermediate variable.

To test that it worked, add the line notify { "The value is ${systemproductname}": } somewhere by itself in one of your pp files and run something like puppetd --test or puppet apply. On my computer, an OptiPlex 990, I see the following get printed to the screen: notice: The value is OptiPlex 990.

So what can you do with it? Well, in my case, I wanted to add a custom kernel option for Optiplex 990s on my network. I feel pretty clever for actually getting this Augeas snippet to work, so I’ll share it here:

augeas {
  "Add reboot=pci to Optiplex 990s to fix reboot issue":
    context => "/files/etc/grub.conf",
    lens    => "grub.lns",
    incl    => "/etc/grub.conf",
    changes => "set title[1]/kernel/reboot pci";
}

This will work on most puppet systems. If you have a newer version of puppet that supports setm, you can set this kernel option for each kernel on the system (please note the spaces after title and between kernel/reboot and pci).

augeas {
  "Add reboot=pci to Optiplex 990s to fix reboot issue":
    context => "/files/etc/grub.conf",
    lens    => "grub.lns",
    incl    => "/etc/grub.conf",
    changes => "setm /files/etc/grub.conf/title kernel/reboot pci";
}

Of course, now the question is, how do we restrict this to just Optiplex 990s? Unfortunately, the augeas type’s onlyif property is only meant to support Augeas statements, so we have to use something else. Happily, Puppet has support for several different conditional types, including inline forms that you could use to support setting different kernel options for different values of $systemproductname. However, for a quick and dirty solution, you can use the case statement. I use it continuously in my puppet configs to check what operating system each one of my clients is running (I have to support both Linux and Macs at work). It will work perfectly here as well. In its least elegant form, just enclose the entire augeas statement with a construction like the following:

case $systemproductname {
  "OptiPlex 990": {
    augeas {
      ...
    }
  }
}

Yup, that’s a lot of indentation. But it works! I hope this helped at least one person feel like a Puppet/Augeas ninja.

Hey, guess what? This works with Mountain Lion (10.8) too. I love you, Mountain Lion.

UPDATE (9/14/2012): I put a lot of redundant lines and options in Mac OS X’s pam.d files that I didn’t need to. I finally was able to find the time to go back and test things more thoroughly during our recent office update to Mountain Lion. Please scroll down to see the updated pam.d files for authorization, sshd, and sudo.

After a lot of fighting, Kerberos support on Mac OS X 10.7 is back. The drawback? Logging in with LDAP at the same time doesn’t always work. I’ve learned a few lessons along the way, though. If you haven’t seen it, my first post, My Mac OS X 10.7 Kerberos Workarounds, describes how to handle setting up keytabs and touches on /etc/pam.d/authorization, but I made a few errors.

Here’s what DIDN’T work (or at least, wasn’t necessary): Adding Apple’s authAuthority attributes to my LDAP schema. Yeah, I know I said to do it, but it was a red herring. It works with or without it. What authAuthority DOES do is replace the need for the pam_krb5.so option default_principal. This option relies on the fact that the user’s principal and username are the same (i.e., username’s principal is in the format username@REALM and their login name is also username). If your environment is different, use authAuthority and delete default_principal from my example pam.d files.

Here’s what does work and what my /etc/pam.d/authorization file looks like:

# authorization: auth account
auth       sufficient     pam_krb5.so use_first_pass default_principal
auth       optional       pam_ntlm.so use_first_pass
auth       required       pam_opendirectory.so use_first_pass nullok
account    required       pam_opendirectory.so

The second line is new, I got the tip from Iowa State University. I’d already recommended adding the option default_principal and still stand by it. It maps your Kerberos principal to its short form (sans realm). This will get you logins and screensaver unlocking for free. To get sudo to work, you’ll have to add something similar to the beginning of /etc/pam.d/sudo like so (use_first_pass in the second auth line is necessary to prevent getting prompted for a password twice when using a local account):

# sudo: auth account password session
auth       sufficient     pam_krb5.so try_first_pass default_principal
auth       required       pam_opendirectory.so use_first_pass
account    required       pam_permit.so
password   required       pam_deny.so
session    required       pam_permit.so

They also gave me another interesting tip, if you’ve ever tried to run kinit from the command line in Mac OS X 10.7, you can foobar future logins and prevent yourself from getting tickets. Apparently there is (or was, maybe it’s been fixed) a bug that would screw up the TGT caches. Run kdestroy, log out, and try again. You can read up on their Kerberos findings here: http://tech.its.iastate.edu/macosx/bugs/lion-pam-kerberos.shtml I don’t think this affects Mac OS X 10.8, but I haven’t tried to replicate it there.

Hopefully, nothing is going to make Kerberos break again, Apple willing.

And finally, if you let users log into machines with SSH, you will want to update /etc/pam.d/sshd to look something like (this was pulled from a 10.8 machine, if your 10.7 machine looks a little different, just focus on the auth lines):

# sshd: auth account password session
auth       sufficient     pam_krb5.so try_first_pass default_principal
auth       optional       pam_ntlm.so try_first_pass
auth       optional       pam_mount.so try_first_pass
auth       required       pam_opendirectory.so try_first_pass
account    required       pam_nologin.so
account    required       pam_sacl.so sacl_service=ssh
account    required       pam_opendirectory.so
password   required       pam_opendirectory.so
session    required       pam_launchd.so
session    optional       pam_mount.so

It occurs to me that I haven’t touched on Apache’s Kerberos support, which is provided by the mod_auth_kerb module. This is the module that lets you restrict access to web pages using Kerberos or store Kerberos tickets for web applications to use. Much like in my previous post, Kerberizing Services in RHEL6, we’ll create a host principal for Apache and install the authentication module that can talk to it.

# kadmin -p username/admin
kadmin: addprinc -randkey -clearpolicy http/hostname.domain.com

Notice I used hostname, not an alias like www. Log into the Apache server and create a keytab for http. We will use an http specific keytab in /etc/httpd/httpd.keytab:

# kadmin -p username/admin
kadmin: ktadd -k /etc/httpd/httpd.keytab http/hostname.domain.com

As a reminder, if you try to use the system’s keytab (by default, usually /etc/krb5.keytab), the apache user account may not be able to read from it. It’s also good practice to only give each service exactly what it needs. On that note, change the ownership and permissions for our new keytab:

# chown apache:apache /etc/httpd/httpd.keytab
# chmod 600 /etc/httpd/httpd.keytab

Now we will install and configure the mod_auth_kerb Apache module. It has its own website, but I’ll outline the basic steps here:

# yum install mod_auth_kerb

Open /etc/httpd/conf.d/auth_kerb.conf with your favorite text editor and make sure the following line is uncommented:

LoadModule auth_kerb_module modules/mod_auth_kerb.so

The next time Apache is restarted, the module will be loaded and web applications that support Kerberos/SASL can use it. The exact process for doing this will vary by application, but to give an example, I’ll discuss the popular phpLDAPadmin package, which we use heavily here at work.

Kerberizing phpLDAPadmin
We get our copy of phpLDAPadmin through the epel repository since we use RHEL 6 on all our machines. As of the time I wrote this, we were using version 1.2.2, which also happened to be the latest version. This matters, since the syntax for its main config file changes from version to version, and when I first began configuring phpLDAPadmin to use Kerberos credentials instead of LDAP passwords, I didn’t realize our config file was from a much older version.

The first thing to do is to somehow get the Kerberos credentials from the user. To do so, we will edit /etc/httpd/conf.d/phpldapadmin.conf and add the appropriate authentication mod_auth_kerb configuration options inside <Directory /usr/share/phpldapadmin/htdocs>:

AuthName "Kerberos Access"
AuthType Kerberos
KrbMethodNegotiate On
KrbMethodK5Passwd On
KrbServiceName http
Krb5KeyTab /etc/httpd/httpd.keytab
KrbSaveCredentials On
KrbLocalUserMapping On
require valid-user

AuthName and AuthType should be familiar if you’ve ever done Apache authentication before. KrbMethodNegotiate is the magic that lets users pass on their existing credentials to the server. The web browser must support this form of negotiation. Firefox does, but not every browser will. If the negotiation fails, KrbMethodK5Passwd is what will prompt a user for a password before continuing. KrbServiceName is the prefix we gave to the http host principals and Krb5KeyTab is the path to the keytab we created. KrbSaveCredentials is what will allow us to pass these credentials on to phpLDAPadmin. require valid-user should also be self explanatory.

You’ll notice I skipped one. KrbLocalUserMapping will strip the realm name from our user id. This is not strictly necessary for phpLDAPadmin, but it’s invaluable for combining Kerberos with LDAP checks, like checking for ldap group membership. Obviously, ldap doesn’t know what to do with user@example.com. But when we use KrbLocalUserMapping, we can do something like this:

AuthLDAPURL ldap://ldapserver.domain.com:389/dc=domain,dc=com?uid?sub?(objectClass=*)
AuthLDAPGroupAttribute memberUid
AuthLDAPGroupAttributeIsDN off
require ldap-group cn=admins,ou=groups,dc=domain,dc=com

Now we’re not just checking if the user is a valid Kerberos user, now we’re actually checking to see if they’re a member of the admins group.

At this point, our Apache configuration is done. The next step is to edit phpLDAPadmin’s config.php file to add the options we need to get SASL/Kerberos to work. In RHEL 6, if you’re using the epel repository’s rpm, this file will be located at /etc/phpldapadmin/config.php. If you scroll down a bit, you’ll see a section with some commented out LDAP and SASL Authentication settings. If you’ve already set up phpldapadmin, you should have done some of this already, but here’s what we use regardless:

$servers = new Datastore();
$servers->newServer('ldap_pla');
$servers->setValue('server','host','ldapserver.domain.com');
$servers->setValue('server','port',389);
$servers->setValue('server','tls',true); // we use TLS and so should you

$servers->setValue('login','auth_type','sasl');
$servers->setValue('sasl','mech','GSSAPI');
$servers->setValue('sasl','realm','DOMAIN.COM');
$servers->setValue('sasl','authz_id_regex','/^uid=([^,]+)(.+)/i');
$servers->setValue('sasl','authz_id_replacement','$1');
$servers->setValue('appearance','password_hash','');
$servers->setValue('login','attr','dn');
$servers->setValue('login','anon_bind',false); // we don't allow anonymous logins

The host and realm settings will need to be configured for your environment. One quick note, setting password_hash to an empty string doesn’t disable hashing, but tells phpLDAPadmin to use the default setting instead. You may wish to set it explicitly for safety’s sake in your environment.

At this point, you can restart Apache and open phpLDAPadmin in your browser. Assuming you have a valid ticket, phpLDAPadmin should load without any password prompts, and the menu on the left hand side should have a line that reads “Logged in as: username”.

If it doesn’t work, there is some debugging you can try. First, you should create a test page protected with Kerberos to test your mod_auth_kerb settings. The full list of options for mod_auth_kerb can be found on their official web page. If that works, then you can go back to testing phpLDAPadmin settings. Make sure you have the latest version (earlier versions did not always have working SASL support). If phpLDAPadmin was not already installed and working on your system, you may want to comment out the sasl settings and replace $servers->setValue('login','auth_type','sasl'); with $servers->setValue('login','auth_type','session'); instead (assuming you have ldap accounts with passwords you can use). This will let you tackle getting it to connect to your LDAP server. There’s a possibility of misconfiguration at each step, and it’s important to go in order.

In RHEL 6.2, at least, SSSD doesn’t always clear its cache for the Kerberos provider. I found this out when I decided to change the group name on our LDAP server. Computers that used straight up LDAP dutifully reflected the change nearly immediately (we use nslcd, so restarting the nslcd service provided a nearly instantaneous update). However, our Kerberos machines were more troublesome, and they refused to reflect the changes, even after restarting nslcd and sssd. Rebooting didn’t help either. Finally, I uncovered the location of the SSSD cache files. To get SSSD to recreate its cache, stop the service and delete the cache files in /var/lib/sss/db/. I deleted all three (cache_default.ldb, cache_kerberos.ldb, and cache_LDAP.ldb). Then all I had to do was start the SSSD service back up again and the problem was fixed. Now id reports the correct group name and directory listings work too.

Update: Some of this has been superseded by my new post on the subject, Mac OS X 10.7: Kerberos is Back. authAuthority does not appear to actually be required, but there is a line that must be added to /etc/pam.d/authorization.

Currently, I have Kerberos on Lion using SSH. To do this, I had to install MacPorts and then install its version of SSH. More work needs to be done. Here I’ve documented what I did to get as far as I have.

Mac OS X 10.7 switched from MIT Kerberos to Heimdal. In order to get this to work, I had to make a few changes to our setup (which is far from mature, unfortunately). If you’ve read my other posts, then you would know that our KDC is running MIT Kerberos on a RHEL 6 server. The same server is also running OpenLDAP. Our goal was to provide Kerberos tickets at login that could be used to SSH into other workstations and protect our NFS shared home directories.

The first thing I had to was to open up the right UDP ports on our KDC (88, 464, and 749). It’s possible to force Heimdal to use TCP instead, but there were a few mixed reports that suggested that it didn’t always work. It seemed easier and more straightforward to let Heimdal use its default settings rather than risk that Apple had made some undocumented tweaks that broke it.

This next part doesn’t seem to be necessary, so I’ve deleted it.
The second thing I had to do was to add authAuthority to our list of mapped LDAP attributes. authAuthority can be found in /etc/openldap/schema/apple.schema on any 10.7 install. The schema file should be added to your OpenLDAP server and the authAuthority attribute type should be uncommented. Using phpldapadmin, I first added authAuthorityObject to my user’s objectClass. Then I added a new attribute, authAuthority and set it to read ;Kerberosv5;;;REALM; (the semicolons are important), replacing REALM with our realm name. Then, using Directory Utility (the easiest way to access it is through the Users & Groups System Preferences pane, but you can find it in /System/Library/CoreServices), I navigated to our LDAP server’s “Search & Mappings” tab, drilled down under Users, and set AuthenticationAuthority on the left side to map to authAuthority on the right side. If AuthenticationAuthority isn’t listed, you can add it. Click on Users (or add it as a Record Type), click add, select Attribute Types, and find AuthenticationAuthority in the list.

A quick note: If you don’t have authAuthority stored in OpenLDAP, but you have /etc/krb5.conf already in place, it’s possible that your network accounts will be unable to log in or use sudo until you do. Deleting /etc/krb5.conf will restore logins for you if you get stuck partway through this process, but it’s better to use a local login until Kerberos is completely set up.

The third second thing to do is to add the krb5.conf file. In previous versions of Mac OS X, you had to name your config edu.mit.Kerberos and hide it in /Library/Preferences/. Lion will still look there for the sake of backwards compatibility, but I didn’t like the idea of naming my Heimdal configuration file by another name. Happily, /etc/krb5.conf is supported (and probably the preferred location going forward). I was able to use my existing krb5.conf file from my RHEL 6 server without any changes. You can read man krb5.conf to make sure your configuration file won’t give you any surprises.

The fourth third thing I had to do was to make sure to edit /etc/pam.d/authorization. This is what the login window uses to fetch your Kerberos tickets. If you open the file, you’ll see that the first line actually calls pam_krb5.so. This replaces some of /etc/authorization‘s functionality. You only need to edit this file if you want to use uids as login names instead of the fully qualified uid@REALM format, but if you append default_principal to the end of the pam_krb5.so line, you will be able to use short names to login, and it will automatically append your default REALM to the end of your login. Here’s an example:

auth       sufficient       pam_krb5.so use_first_pass use_kcminit default_principal

Further testing seemed to reveal that you might need to remove use_kcminit from the same line in /etc/pam.d/authorization:

auth       sufficient     pam_krb5.so use_first_pass default_principal

At this point, if you’re feeling brave, run killall opendirectoryd if you haven’t already, log out, and log back in as a network user account. If you have already correctly configured network user accounts with OpenLDAP, everything should just work, and when you open Terminal and run klist or open Ticket Viewer (buried in /System/Library/CoreServices), you should see a ticket for your user account with a valid expiration date!

As a final step, if you need host principals for your workstations, I had some interoperability problems between Heimdal and my MIT Kerberos KDC. The way I worked around it was to create a keytab with kadmin on a RHEL 6 workstation, copy it to the new computer, and use ktutil on the destination Mac to append it to the default /etc/krb5.keytab file that comes with Mac OS X. (I managed to really break a Lion install a while back by deleting its keytab file, and rather than risk doing that again, I decided to let the existing principals remain. Recently, however, just copying the keytab file directly from a Linux machine to /etc/krb5.keytab seems to work.) Here’s essentially what I do now for new Mac OS X computers:

1. I log into a RHEL 6 workstation running MIT Kerberos, and log into kadminwith my admin principal:

   # kadmin
   kadmin: ktadd -k computername.keytab host/computername.domainname
   kadmin: ktadd -k computername.keytab nfs/computername.domainname

   This takes care of SSH and NFS. If your network needs other Kerberos principals on workstations, add them to the keytab at this time.

2. I copy the file (in my case, using SSH) to the newly installed Mac. Then I (optionally) run ktutil with sudo to merge the two keytab files together. Otherwise, just overwrite /etc/krb5.keytab.

   # ktutil copy computername.keytab /etc/krb5.keytab

3. Now that the host principals have been safely merged, I can delete the temporary keytab I’ve created.

Lion introduced OpenDirectory as a replacement for DirectoryService. The configuration looks practically identical in the GUI, but the output plist files are different enough that you have to recreate them. I use a custom LDAP mapping, so to help me remember my settings, I had my old DSLDAPv3PlugInConfig.plist open in a text editor while I went through the tedious process of recreating it. Not everyone has to make these mappings and I know a lot of people can get away with using the standard RFC mapping. Lucky me.

If you have to create a custom mapping, but haven’t done it before, it’s straightforward enough, but easy to leave things out by mistake. In the Search & Mappings tab, you are mapping LDAP attributes to OpenDirectory attributes. On the left side of the screen, you must add high level Record Types first, then populate them with Attribute Types. Then, on the right side of the screen, you add the LDAP names, like uid or userPassword. If you have an ldif file of a user on your LDAP server as an example, or use a transparent LDAP editor like phpldapadmin, the LDAP names will be easy to find. Then it just becomes a matter of discovering what Mac OS X calls each thing.

In earlier versions of Mac OS X, I copied DSLDAPv3PlugInConfig.plist and SearchNodeConfig.plist to /Library/Preferences/DirectoryService and told puppet to restart the com.apple.DirectoryServices service. Super straightforward! This left me totally unprepared for a truly horrible snafu when trying to script our LDAP settings in Mac OS X Lion the same way.

So in Lion, I copied the correct files (Search.plist, Contacts.plist, and ldap.domainname.plist) into the correct places, but when I told puppet to restart the com.apple.opendirectoryd service, the service fell over and refused to start back up. Worse, now that OpenDirectory was dead and no longer mapped my user name to my user id, suddenly I could’t use sudo anymore or run anything with administrator privileges. Even single user mode couldn’t help me, and deleting the entire contents of /Library/Preferences/OpenDirectory didn’t seem to help. Next I tried reinstalling, but apparently that didn’t overwrite the correct files either. That’s right, I got stuck having to FORMAT my drive just to get a working copy of Lion back.

So where did I go wrong? As it turns out, after reading some online documentation more closely, you are SUPPOSED to run killall opendirectoryd and avoid ever touching its launchctl file at all. I was skeptical, but I tried it and suddenly, it WORKED. From there, it was trivial to script this in puppet:

exec {
  "com.apple.opendirectoryd":
    command     => "killall opendirectoryd",
    refreshonly => true;
}

file {
  "/Library/Preferences/OpenDirectory/Configurations/LDAPv3":
    owner  => root,
    group  => wheel,
    mode   => 750,
    ensure => directory;
  "/Library/Preferences/OpenDirectory/Configurations/Search.plist":
    owner => root,
    group => wheel,
    mode  => 600,
    ensure => present,
    notify => Exec["com.apple.opendirectoryd"],
    source => "puppet:///modules/ldap/Search.plist";
  "/Library/Preferences/OpenDirectory/Configurations/Contacts.plist":
    owner => root,
    group => wheel,
    mode  => 600,
    ensure => present,
    notify => Exec["com.apple.opendirectoryd"],
    source => "puppet:///modules/ldap/Contacts.plist";
  "/Library/Preferences/OpenDirectory/Configurations/LDAPv3/ldap.domainname.plist":
    owner  => root,
    group  => wheel,
    mode   => 600,
    ensure => present,
    notify => Exec["com.apple.opendirectoryd"],
    source => "puppet:///modules/ldap/ldap.domainname.plist"
}

At work, I applied the RHEL 5 Draft STIG to some of our systems in an effort to increase our security. (STIGs are security checklists, and they’re available for a multitude of operating systems and devices. Unfortunately, they are frequently out of date, hence why I’m applying a RHEL 5 STIG to a RHEL 6 network.) A week later, and a routine kernel update brought our network to its knees. The guilty party? /usr/bin/ldd‘s permissions, which the draft STIG required be set to 000.

Apparently, when you update a kernel package in RHEL (and this may be a shortcoming of RPM packages in general, since root should be able to get around this problem), it tries to run /usr/bin/ldd. If you remove all access to ldd by setting its permissions to 000, the kernel does not install correctly (even though yum reports no errors) and you will get a Kernel Panic screen on your next boot. I got a few variations of the same message (one said “VFS: Unable to mount root fs on unknown-block(0,0)” and another one said “Kernel panic - not syncing: No init found. Try passing init= option to kernel.”). When I first encountered the problem, it was a week after I’d applied the STIG, so my gut reaction initially was to say it was RHEL’s fault. I didn’t really believe that, but I couldn’t see what else it could be.

It wasn’t until many hours later that I tried to run mkinitrd manually out of desperation. It started spitting out error messages about ldd. Suddenly it dawned on me what had happened. I fixed the permissions and tried rebooting. Still nothing. Then it occurred to me that ldd was probably only getting run during the initial kernel INSTALL. This time, I booted off the old kernel, ran yum reinstall kernel and rebooted. SUCCESS!

Unfortunately, there isn’t a good way to detect that a kernel update is about to be applied, so it would be hard to script resetting the permissions before an install. I’m going to recommend that administrators skip this STIG setting for now. Bricking your system is too high a price just to be compliant, and I’m unconvinced that changing ldd‘s permissions provides any security benefit whatsoever.

We’re stuck with an old version of puppet on our Macs because our puppet server is running RHEL 6 (surprise) which is stuck at 2.6.14. My previous attempts to bridge major version differences have failed miserably. Now I just keep them in sync and sigh at all the awesome features I don’t get to play with just yet.

I don’t know how we didn’t run into this problem in the past (perhaps it was a change on puppet’s part), but during my Lion testing, I noticed that puppet was trying to load launchctl jobs without the -w flag. In particular, Cups and SSH were refusing to load. After some Googling, I discovered that puppet didn’t know that new versions of Mac OS X no longer stored the Disable key inside the launchctl plist file in /System/Library/LaunchDaemons. Instead, Disable keys are stored in /var/db/launchd.db/com.apple.launchd/overrides.plist.

It looks like newer versions of puppet have a fix, but unless that fix gets back ported, I’m stuck finding my own solution. Enter PlistBuddy, a simple (and somewhat stupid) tool for editing plist files. Here’s an example of how to get the Disable key settings for Cups and SSH (I apologize for my web design laziness; if your window isn’t wide enough, the text will run off the screen):

# /usr/libexec/PlistBuddy -c "Print :org.cups.cupsd:Disabled" /var/db/launchd.db/com.apple.launchd/overrides.plist
# /usr/libexec/PlistBuddy -c "Print :com.openssh.sshd:Disabled" /var/db/launchd.db/com.apple.launchd/overrides.plist

Please note the colons (:) in the command we send to PlistBuddy. If you open the /var/db/launchd.db/com.apple.launchd/overrides.plist file, you’ll see that it’s in XML and that it’s a dictionary with many keys. Each key in that dictionary is the name of a launchctl job. At the highest level, you can run something like:

# /usr/libexec/PlistBuddy -c "Print :" /var/db/launchd.db/com.apple.launchd/overrides.plist

This prints the dictionary keys and their values (which, as you might have noticed, are also dictionaries). The following will let you see the value of just one dictionary key, “com.openssh.sshd”:

# /usr/libexec/PlistBuddy -c "Print :com.openssh.sshd" /var/db/launchd.db/com.apple.launchd/overrides.plist

This prints out the dictionary stored inside the com.openssh.sshd key. And lastly, the original command I gave you will give you the value of the key inside that dictionary:

# /usr/libexec/PlistBuddy -c "Print :com.openssh.sshd:Disabled" /var/db/launchd.db/com.apple.launchd/overrides.plist

A value of true means the service will not start, and a value of false means that the service can start. PlistBuddy can also set values. The syntax is the same, but instead of using Print, you pass it the Set command. For something simple, like setting something to true or false, this is very straightforward:

# /usr/libexec/PlistBuddy -c "Set :com.openssh.sshd:Disabled false" /var/db/launchd.db/com.apple.launchd/overrides.plist

OK, now that I’ve walked you through how PlistBuddy works (you can read about other supported commands in man PlistBuddy), I’ll show you how to set the value in puppet, using the exec type. Here’s an example for SSH:

exec {
  "Turn off Disabled key for SSH":
    command => "/usr/libexec/PlistBuddy -c 'Set :com.openssh.sshd:Disabled false' /var/db/launchd.db/com.apple.launchd/overrides.plist",
    onlyif  => "/usr/libexec/PlistBuddy -c 'Print :com.openssh.sshd:Disabled' /var/db/launchd.db/com.apple.launchd/overrides.plist | grep -q true";
}

Here, the command simply sets the Disabled flag to false, provided that it’s currently set to true. Obviously, this would lend itself very well to scripting. You could replace “:com.openssh.sshd:Disabled” with a puppet variable and call it for each service you wanted to enable. And if you wanted to be sure that the Service resource loaded correctly the first time, you could use a require line to tell puppet to try running the exec first:

Service {
  "com.openssh.sshd":
    ensure  => true,
    enable  => true,
    hasstatus => true,
    require => Exec["Turn off Disabled key for SSH"];
}

I hope this helps someone else. I wasted too much time on this trying to be clever. I’m looking forward to using a version of puppet that has a patch for this built into it.