So much to say, so little time to say it!

Thursday, June 18th, 2009

I’ve been working on a lot of fun things both personally and for my company. However, I haven’t prioritized blogging about it. Maybe I should add a GTD recurring task to ‘blog about what’s on my plate.’

Things I’ve been focusing on:

Learning more about agile methods, especially as it pertains to to dealing with teams that have operational requirements as well.

Working more with my OpenSolaris NAS. Suprisingly, one of my most popular ever blog articles.

Playing lots of music: I’m playing bass and guitar in my cover band’s summer concert this weekend. (Can’t wait!)

Planning an epic road trip for the summer: 5 weeks up the West Coast, starting in July.

And, of course, loving my little 5 month old!

So, more blogs to come.

Great meal at Providence

Wednesday, April 29th, 2009

Last night we had a delicious and amazing dinner, celebrating Amber’s 27th birthday, at Providence.

It was a ‘5×5 Chefs Collaborative’, meaning 5 local chefs (and 1 guest chef, Alex Stratta) hung out together and handled one course each.

They gave us a menu to take home that I’ve attached. All the chefs signed it, which was a nice touch I’d never seen before.

The menu: Providence 5×5 Signed Menu.pdf

Every course was incredible. Our favorite courses were probably the amuses, the hamachi sashimi, and the best testament to the power of sous vide I’ve had in a long time — Alex Stratta’s duck.

Yum.

Managing volatile files with puppet

Wednesday, April 15th, 2009

I’m managing Linux HA (heartbeat2) from Puppet and I had a problem.
There is a file called ‘cib.xml’ used by heartbeat which I needed to manage. For a variety of boring reasons, when you make changes to this file, you must have a number in it which is the highest the
app has ever seen.

To make things easy, I use the current timestamp for this, as I’ve never seen that number go backwards.

Ok, not the normal kind of thing you do with Puppet, but it can be done. We just need to know what time it is in the template.

Thankfully Puppet lets us define custom facts, so here we go…
in the module’s plugins/facter directory, I created ‘epoch_time.rb’.

Facter.add("epoch_time") do
        setcode do
                Time.now.to_i
        end
end

Then I could use that in templates/heartbeat/cib.xml I said

 <cib admin_epoch="<%= epoch_time %>">

Well, that works fine, except for one little issue.

When puppet manages a file it does this:

  1. Renders out the templates and takes an md5sum of that
  2. Sends that to the client, which checks it’s own md5sum
  3. If the sums are different, the client overwrites the file with the old one
  4. The client then updates anything ’subscribed’ to that file (for example, restarting a service.)

So since puppet runs about every 30 minutes, that’s not good. I don’t want to be reloading the CIB database every 30 minutes. That’s bad.

This is what we get:

debug: File[/etc/ha.d/ha.cf]/checksum: Initializing checksum hash
debug: File[/etc/ha.d/ha.cf]: Creating checksum {md5}18530322762561ce59f1d414340b4c43
debug: File[/etc/ha.d/cib.xml]/checksum: Initializing checksum hash
debug: File[/etc/ha.d/cib.xml]: Creating checksum {md5}a02f0ca8a3cce64d7913faa3268e530b
debug: File[/etc/ha.d/cib.xml]/content: Executing 'diff /etc/ha.d/cib.xml /tmp/puppet-diffing20090415-3486-ska653-0'
1c1
< 
---
>  
debug: File[/etc/ha.d/cib.xml]: Changing content
debug: File[/etc/ha.d/cib.xml]: 1 change(s)

So here was my solution, and as inelegant as it is, it pretty much works.

  1. Manage a file called ‘cib.xml-puppet’ instead
  2. In this file, set admin_epoch=”0″
  3. When THAT file changes, do a chained action which sets the epoch time correctly, and writes out the actual cib.xml file
  4. Run the command to reload the CIB database in that and only that case.

So the puppet code to make this happen:

        exec { "add-epoch-cib-xml":
                command => "sed 's/admin_epoch=\"0\"/admin_epoch=\"${epoch_time}\"/' /etc/ha.d/cib.xml-puppet > /etc/ha.d/cib.xml && cibadmin -R -x /etc/ha.d/cib.xml",
                path => ["/bin", "/usr/sbin/"],
                subscribe => File["/etc/ha.d/cib.xml-puppet"],
                refreshonly => true,
        }
        file { "/etc/ha.d/cib.xml-puppet" :
                type => 'file',
                ensure => 'present',
                content => template("ha_nfsroot/heartbeat/cib.xml");
        }

So I still get to use my very first custom fact (it gets interpolated into the ’sed’ command) but I also don’t run that terrifying update every time.

If any puppetmasters or Linux HA gurus want to tell me exactly how I’m doing it wrong I would love to hear that. This certianly doesn’t feel “elegant.”
However, since it feels “functional”, functionality trumps elegance for today.