Jul201006

Mike

0 comments

Developing Qnotifier plugins

Qnotifier is a combination of a daemon that runs on your Unix system (Linux or OSX), and an iPhone/iPad app to provide a dashboard-like view of data collected from your system. It can report text-based information, time-based graphs, or alerts which can be delivered to you in a variety of ways—for example, when your CPU usage gets too high. Qnotifier is free right now while in beta, but no word yet on whether it will switch to a paid model in the future.

One of the things I like about Qnotifier is that the daemon is a Ruby gem, and supports user-developed plug-ins to collect and report new kinds of information. I just finished setting up a Squid cache to sit in front of a Rails site to improve performance, and decided to see what it would take to add Squid statistics to my Qnotifier dashboard. It turns out, getting the Squid statistics was the hard part—delivering them to Qnotifier was easy!

Getting Squid statistics

I couldn’t find much information on how to retrieve statistics. Every web page said to install and use the Squid cachemgr.cgi tool to get a GUI into viewing stats. This wasn’t what I wanted, but it got me there eventually. The CGI is compiled C code, and quick look through the source gave some hints. Using tcpdump to observe it in action gave away the answer: send the following request to the running Squid server, and get back a ton of information:
GET cache_object://localhost/info HTTP/1.0

Writing the Qnotifier plugin

Qnotifier was just released a week ago, fairly stealthily, and thus there’s no documentation at all yet. Even looking through the plugins shipped with the package, half of them are non-functional yet (apache, iostat, etc.).

Fortunately, it was pretty easy to figure out the general plugin architecture from reading the working plugins (Ruby, system, Passenger, etc.):

  • Create a new plugin file in /var/lib/qnotifier/plugins
  • Create a new Class, and inherit from Qnotifier::Plugin
  • Collect your metrics in whatever way is appropriate.
  • Call stat(key, message) to deliver text-based metrics to the “Stats” dashboard in the app.
  • Call report(key, val) to deliver one time-series data point for a graph in the “Charts” dashboard.
  • Call alert(key, message) to send a real-time alert, and reset_alert(key, message) to clear one.

Here’s the short Ruby plugin I wrote to retrieve the Squid statistics (using netcat) and push them to Qnotifier. This is an hour’s proof-of-concept work, and just gathers a couple metrics without error checking. If Qnotifier comes out of beta and is free (or cheap), I’ll clean it up some more since I see a lot of potential here.

module Qnotifier
  class Squid < Qnotifier::Plugin

    def initialize
      @defaults = {
      }
    end
   
    def main
      status = `echo "GET cache_object://localhost/info HTTP/1.0\n" | nc -q 2 localhost 3128`

      if status
        data = Hash.new
        # Convert all the key: value lines into a Hash
        status.split("\n").each do |line|
          s = line.split(':', 2)
          data[s[0].strip] = s[1].strip if s.length == 2
        end

        # How much space is used in the disk cache
        # Looks like "7126 KB"
        d = data['Storage Swap size'].split(" ")
        disk_space_used = d[0].to_i / 1024 if d[1] == 'KB'
        disk_space_used = d[0].to_i if d[1] == 'MB'

        # What's the 5-minute request cache hit rate
        # Looks like "5min: 93.5%, 60min: 97.4%"
        req_hit_rate = data['Request Hit Ratios'].match(/5min: ([^%]+)/)[1]

        # Send data to the service
        report("Disk cache used (MB)", disk_space_used)
        report("Request cache hit ratio %", req_hit_rate)
        stat("R hit rate", data['Request Hit Ratios'])
        stat("B hit rate", data['Byte Hit Ratios'])
      else
        Qnotifier.log.error "Can't access squid status"
      end
    end
  end
end

Jun201002

Mike

0 comments

Confluence Ruby API

At work, we use the most excellent Confluence wiki system by Atlassian for all of our shared documentation. We also use it for weekly status reports, which are based on a template that needs some minor changes each week before being created.

Creating these weekly pages for people to fill in used to be a manual process—copying the template to a new wiki document, then editing it to make some pre-defined changes (such as the date)—until I learned about a Ruby gem for the Confluence API. Now a small Ruby program run weekly does the work for me:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
#!/usr/bin/ruby

require 'rubygems'
require 'confluence4r'

# Some variables specific for my Confluence setup
url = 'https://your.confluence.wiki.com/'
user = 'jamesbond'
password = '007'
wikispace = 'statusreports'
template = 'Status for week ending YYYY-MM-DD'

# Find the date of the next Friday
friday = Date.today
friday += 1 while friday.wday != 5

# Log in to Confluence and retrieve the template page
server = Confluence::Wiki.new(url)
server.login(user, password)
page = server.getPage(space, template)

# Create a new page based on the template, changing the
# date placeholder with this Friday's date
newpage = Hash.new
newpage['space'] = space
newpage['title'] = template.gsub(/YYYY-MM-DD/, friday.to_s)
newpage['content'] = page['content'].gsub(/YYYY-MM-DD/, friday.to_s)
newpage['parentId'] = page['parentId']

# Save the new page
server.storePage(newpage)
server.logout()

Resources:
* Confluence by Atlassian
* confluence4r gem on github
* confluence4r documentation by Atlassian

Mar201001

Mike

1 comments

Javascript date selector in Rails

Many projects I work on require a user to select dates while entering information into a form. Rails makes it trivial to create date selectors and save the results into models via the date_select form helper. Here’s what it looks like by default:

Functional, but hardly friendly to require the user to manipulate three different controls to select a date. Searching the web I found a few different pop-up-calendar JavaScript apps, and settled on using Brian McAllister’s Unobtrusive Date-Picker Widget via Brian landau’s Rails plugin.

I’ve used this successfully for the past year, but noticed this message on the plugin’s home page recently: The latest release of the Unobtrusive Date-Picker Widget JavaScript library makes this plugin unnecessary so I’ve decided to no longer maintain it. Using the date-picker without needing a plug-in sounded great to me! Unfortunately, I couldn’t find any references or instructions on how to use the new (version 5) date-picker with Rails, and there was clearly no built-in support within new new date-picker.

I set out to solve this problem and this is the solution I came up with. The previous plug-in version required changes to my Rails applications’ views in order to activate the date-picker. Reading through the documentation for the new version 5 date-picker JavaScript app, it quickly became obvious that I’d need to handle the date-picker activation completely in JavaScript. This has advantages (the date-picker is truly unobtrusive, since no Rails app changes are needed) and disadvantages (flexibility is lost since the Rails app can’t easily configure the date-picker).

The date-picker needs to bind to the existing date_select form elements in order to function. This example from the documentation demonstrates, with three form elements with DOM IDs of inp1, inp2, and inp3:

datePickerController.createDatePicker({
    formElements:{
        // "inp1" represents the day part
        "inp1":"d",
        // "inp2" represents the month part
        "inp2":"m",
        // "inp3" represents the year part
        "inp3":"Y"
    }
});

Rails view helpers create form element IDs automatically. For example, a form for a “person” model with a “firstname” input would have an ID of person_firstname. Even though dates are stored as a single field in models, the date_select helper creates three form elements (year, month, and date), which are re-combined when saving to the model. Each of these elements’ IDs are generated as above, but with an identifier (_1i, _2i, or _3i) appended to differentiate the three fields. My first thought was to retrieve the computed IDs in my view code, and pass them to the JavaScript. Looking through the DateHelper source code yielded some promising hints such as the input_id_from_type method, but I eventually gave up as nothing seemed easily accessible from public-facing methods. Instead I decided to rely on Rails’ current convention, and search for the _1i identifier across all form select elements in the page.

$(document).ready(function() {
  // Rails creates date_select ids as
  // #{model}_#{attribute}_#{field}i
  // where #{field} is a digit representing
  // each element in the date/time.
  $("select[id$='_1i']").each(function() {
    var id1 = this.id; // e.g. person_birthday_1i
    var id2 = this.id.replace(/1i$/, "2i");
    var id3 = this.id.replace(/1i$/, "3i");
    var eles = {};
    eles[id3] = "j";
    eles[id1] = "Y";
    eles[id2] = "n";
    datePickerController.createDatePicker({formElements: eles});
  });
});

The above snippet from application.js requires jQuery and does the following:

  • Finds all select elements that have an ID ending in _1i. These represent individual date_select instances on the page (in case a page has multiple dates to be selected).
  • For each one, computes the IDs of the three components of the date_select (_1i, _2i, and _3i)
  • Creates a JSON configuration array telling the date-picker which form element describes each part of the date, and in what format. See the Date Formats section of the version 5 date-picker documentation for more details.
  • Instantiates the date-picker

Here’s what the date selection form looks like with the above code in place:

Of course, the magic happens when the user clicks on the calendar icon which now appears to the right of the existing date selection elements:

To get all this working in your project, you’ll of course need to download the date-picker files from Brian McAllister’s site, move the files into your Rails app’s /public directory, and include them with something like the following:

<head>
  <meta http-equiv="content-type" content="text/html;charset=UTF-8" />
  <title>People: <%= controller.action_name %></title>
  <%= stylesheet_link_tag 'scaffold' %>
  <%= stylesheet_link_tag '/datepicker/stylesheets/datepicker.css' %>
  <%= javascript_include_tag 'http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js' %>
  <%= javascript_include_tag '/datepicker/lang/en.js' %>
  <%= javascript_include_tag '/datepicker/javascript/datepicker.packed.js' %>
  <%= javascript_include_tag 'application.js' %>
</head>

Feb201024

Mike

4 comments

Home automation via iPhone

When we had our house built a little over 10 years ago, we had a home automation system installed, which included a Lutron Homeworks lighting control system. This meant every light circuit in the house was wired to Lutron’s controller, which could be programmed with their custom Windows application.

The installation company distributed multi-button keypads throughout the house and programmed them to do interesting things, like turning on a dim path of lights from the bedroom to the kitchen, or the ever-popular “30 seconds from now, turn off all lights in the house” button we use when going out or going to bed. One requirement I had was that they leave me with a copy of the application so I could re-program the keypads myself.

I quickly grew dissatisfied with being limited to the keypads, so did some digging around and discovered the documentation for the RS-232 command set used to program the keypads. I hooked up the serial port of the Lutron controller to my Linux server in the basement and quickly whipped up some HTML+Javascript to show me the status of all the lights in the house and let me change the level of any of the lights. Since we’re a very laptop-enabled household, this meant no more getting up off the couch to adjust the lights! There’s a generally-accepted belief that programmers are lazy, since we often write programs to do things for us, rather than do the things ourselves.

Now that we’re also an iPhone-enabled household, I decided it was time to get with the times. After all, it’s been almost 10 years since my original web-based version. I looked into several UI frameworks for creating iPhone web apps, and decided to go with jQTouch. After one evening I was up and running with an iPhone web app that would let me control the lights The next evening I had the app updated to query the lighting system to display the status of the lights in the house. Touching one of the lights brings up the iPhone radial selector to change the brightness of that light. Touching done sends a command to the Linux server with the RS-232 connection to the Lutron controller.

With jQTouch, the HTML and CSS were very straightforward, and only a little bit of JavaScript glue was needed to bring in the interactivity.

Here’s a bit of the Haml for the main page of the web app:

  %body
    #home
      .toolbar
        %h1 Lighting
      %form
        %h2 Main Floor
        %ul.rounded.lights
          %li
            .title Kitchen
            %select{:id => 'z_01_04_01_08_03'}
              %optgroup{:label => "Kitchen"}
                %option{:value => '0'} 0%
                %option{:value => '20'} 20%
                %option{:value => '40'} 40%
                %option{:value => '60'} 60%
                %option{:value => '80'} 80%
                %option{:value => '100'} 100%
          %li
            .title Breakfast
            %select{:id => 'z_01_04_01_08_04'}
              %optgroup{:label => "Breakfast"}
                %option{:value => '0'} 0%

And the HTML which is delivered to the iPhone after Haml conversion:

  <body>
    <div id='home'>
      <div class='toolbar'>
        <h1>Lighting</h1>
      </div>
      <form>
        <h2>Main Floor</h2>
        <ul class='rounded lights'>
          <li>
            <div class='title'>Kitchen</div>
            <select id='z_01_04_01_08_03'>
              <optgroup label='Kitchen'>
                <option value='0'>0%</option>
                <option value='20'>20%</option>
                <option value='40'>40%</option>
                <option value='60'>60%</option>
                <option value='80'>80%</option>
                <option value='100'>100%</option>
              </optgroup>
            </select>
          </li>
          <li>
            <div class='title'>Breakfast</div>
            <select id='z_01_04_01_08_04'>
              <optgroup label='Breakfast'>
                <option value='0'>0%</option>

jQTouch’s CSS turns the unordered list with rounded class into a pretty iPhone screen, and its JavaScript enables effects like moving between screens (not used in this example).

The JavaScript (jQuery) glue I mentioned earlier consists of two parts. First, when the web application loads, it makes an AJAX query to the Linux server and gathers information about the lights, then sets the initial state of the select form fields:

// Populate the data
$(document).ready(function() {
  $.getJSON('/automate/get_lutron.php', function(data) {
    $.each(data, function(key, val) {
      var v = Math.round(parseFloat(val)/20)*20;
      $("#z_" + key).val(v);
    });
  });
});

The Lutron system identifies each light with a 5-part numeric string, like “02_04_01_02_01″. In order to do as little data transformation as possible, I used the same string as the selector IDs in the form (with z_ prepended since XHTML IDs can’t start with a number). Since I’m using 20% brightness granularity, a little math needs to be done for lights that are set to other values. All the lights have manual analog dimmer switches, so a light might be set to any value from 0-100% the old fashioned way (you know, walking over to the switch and physically moving the slider).

The second part of the JavaScript glue listens for changes to any of the selectors in the web app, and then sends the new value to the Linux server’s Lutron RS-232 interface:

// Lutron SELECT elements get wired to AJAX
$(document).ready(function() {
  $('select').change(function() {
    id = this.id.substring(2).replace(/_/g, '.');
    val = $(this).val();
    var command = 'fadedim,' + val + ',1,0,' + id;
    $.get('/automate/send_lutron.cgi', {cmd: command});
  });
});

The end result works beautifully, and now it’s even easier to turn off that forgotten kitchen light after settling down with popcorn in the family room!

Feb201021

Mike

0 comments

LRR.org Version 5

This month I completed a major software development and web design milestone: launching a completely re-designed and re-built web site for Labrador Retriever Rescue. I’ve been the webmaster for LRR for around eight years, and this is my fourth major revision for the site, both from a design and software standpoint. This revision took about a year to achieve, and I’m thrilled with the end result.

LRR.org Version 5: February 2010---

I had a several high-level goals in mind during the process.

  • Make the site more attractive and modern looking
  • Organize the content better and make navigation easier and more intuitive for visitors
  • Make the page URLs more user-friendly (for example: http://lrr.org/reading-room/why-adopt)
  • Create a simple but powerful administrative back-end to allow volunteers to easily add and update content on the site
  • Develop the software to be easily extendable with new features in the future

Many of the goals were meant to address shortcomings of the fourth generation of the web site, which was live from early 2007 until early 2010. Version 4 was developed as a custom wiki application in PHP. I decided to develop Version 5 with Ruby on Rails in order to put some structure around the code for future extensibility, abstract out database record management, and to stretch myself to learn new programming techniques.

While the publicly viewable web site is what most people see, the more interesting code lies in the administrative interface that only Volunteers see. Some of the many “behind the scenes” features are:

  • Role-based user accounts allowing specific volunteers to edit pages, dogs, users, etc.
  • An AJAX group calendar for volunteers and foster homes to record their availability
  • A multi-faceted interface for tracking dogs, including their photos, status, foster home journals, stories from adopters, etc.
  • An interface for adding and editing web pages using the Markdown language
  • Simple interface for adding news updates to the home page
  • Interactive image uploader with feedback, and JavaScript image cropping functionality

The experience was very challenging at times, both from a programming and visual design standpoint, and there’s still much left to be done now that the site is launched. I plan to extract a few learning experiences from the project for future blog posts. Until then, enjoy a trip “back in time” via a look at some older LRR.org iterations. Notes: Versions 1 and 2 were created before I became a volunteer and started managing the web site; the Version 2 picture is missing its navigation menu images; many thanks to The Internet Archive’s WaybackMachine.

LRR.org Version 4: January 2007---February 2010

LRR.org Version 3: September 2001---January 2007

LRR.org Version 2: March 2000---September 2001

LRR.org Version 1: December 1996---March 2000

Feb201001

Mike

4 comments

Automatically activating a form field with JavaScript

I’m a firm believer that software developers should do everything possible to make the end-user experience as simple as possible—in other words, make the software do the hard work so I don’t have to. You probably come across negative examples of this all the time without thinking twice: the web form that requires you to enter a date in a specific format; the phone number field that complains when you insert hyphens in your phone number, and the other one that complains when you don’t insert hyphens. Let the user enter their information any way they want (within reason), and let the software figure it out!

Let’s start off easy, with a short code snippet that demonstrates a very easy way to improve the usability of a web site:

1
2
3
4
// Activate form input that has class of "focus"
$(document).ready(function() {
  $('.focus').focus();
})

Normally, web forms are not activated after a page is loaded—the user needs to mouse into the form and click in the first text box before they can begin typing. Including the above simple bit of jQuery JavaScript in my application lets me add the “focus” class to any form field on any page, and that field will automatically be activated when the page loads:

1
2
3
<form method="post">
  <input class="focus" type="text" name="search">
</form>

It’s a very small gain, but many small gains like this can significantly improve the overall end-user experience on the web.

Jan201025

Mike

0 comments

Welcome to The Last Pixel

I’ve been maintaining a personal web site since May 19, 1993. At that time, I served my “home page” via a CERN HTTPd server I’d set up for my job at the State University of New York at Stony Brook, and NCSA’s Mosaic was the only web browser in town.

The intent of The Last Pixel is to provide a new home for my technology and software development posts, so that my personal blog can remain just that—personal. On The Last Pixel you can expect to find articles on software development, systems administration practices, management insights, and a variety of other technology-related topics. In addition to providing me with an outlet for these thoughts, I also hope that some of you find something useful here!