Setting a custom queryset for a ModelChoiceField in init()

Given the following form:

class ListReqStep1(forms.Form):
    group = forms.ModelChoiceField(
        queryset=IETFWG.objects.all().filter(status=IETFWG.ACTIVE).
            select_related(depth=1).order_by('acronym.acronym'),
        required=False, empty_label="-- Select Working Group")
    domain_name = forms.ChoiceField(choices=DOMAIN_CHOICES,
        required=False, widget = forms.Select, initial='ietf.org')
    list_to_close = forms.ModelChoiceField(
        queryset=ImportedMailingList.objects.all(),
        required=False, empty_label="-- Select List To Close")

The list_to_close choices should really depend on the value of the domain_name field (i.e., the lists that are hosted at that domain). The classic way to handle this is to set the choices in __init__(). However, with a ModelChoiceField, there's a twist. The field's choices are a QuerySetIterator that iterates over the queryset when needed (deferring the query until it's definitely needed). Unfortunately, if you just change the form element's queryset in __init__(), that changes what the form field's choices will return, but not what the widget will use. The obvious code:

    def __init__(self, *args, **kwargs):
        super(ListReqStep1, self).__init__(*args, **kwargs)
        # Base the queryset for list_to_close on the initial value
        # for the domain_name field.
        self.fields['list_to_close'].queryset = 
               ImportedMailingList.choices(self.initial.get('domain_name','ietf.org'))

simply results in the original queryset being used, since even though you've changed the queryset, the widget has a copy of the original QuerySetIterator. To get the right choices, you have to give them to the widget again - this seems very magic, but works:

    def __init__(self, *args, **kwargs):
        super(ListReqStep1, self).__init__(*args, **kwargs)
        # Base the queryset for list_to_close on the initial value
        # for the domain_name field.
        self.fields['list_to_close'].queryset = 
               ImportedMailingList.choices(self.initial.get('domain_name','ietf.org'))
        # This is necessary after changing a ModelChoiceField's
        # queryset.
        self.fields['list_to_close'].widget.choices =
             self.fields['list_to_close'].choices

Custom radio field rendering with django newforms

I'm trying to replicate the look and feel of an existing site, which has a grouping of radio buttons that isn't easy to replicate with the built-in django methods. To really replicate it I need to render each button seperately. There's a class deep inside the RadioButton widget that acts as a list of buttons, which would be perfect, but there's no accessor for it built into the current forms. Luckily, it's easy to get to.

class ListReqStep1(forms.Form):
    mail_type = forms.ChoiceField(choices=(
        ('newwg', 'Create new WG email list at ietf.org'),
        ('movewg', 'Move existing WG email list to ietf.org'),
        ('closewg', 'Close existing WG email list at ietf.org'),
        ('newnon', 'Create new non-WG email list at selected domain above'),
        ('movenon', 'Move existing non-WG email list to selected domain above'),
        ('closenon', 'Close existing non-WG email list at selected domain above'),
        ), widget=forms.RadioSelect())
    def mail_type_fields(self):
        field = self['mail_type']
        return field.as_widget(field.field.widget)

This means that inside the form, I can render each radio button at will using, e.g.,

<h2>Working Group Fields</h2>
{{ form.mail_type_fields.0 }}<br>
{{ form.mail_type_fields.1 }}<br>
{{ form.mail_type_fields.2 }}<br>
<h2>Non-Working Group Fields</h2>
{{ form.mail_type_fields.3 }}<br>
{{ form.mail_type_fields.4 }}<br>
{{ form.mail_type_fields.5 }}<br>

This also gives me a handle to change the rendering of the actual <input> elements. I needed some javascript, but the RadioSelect widget doesn't pass its attrs all the way down to the input elements. This part is a little ugly, but I used regexps to insert an attribute into the rendered element:

    def mail_type_fields(self):
        field = self['mail_type']
        # RadioSelect() doesn't pass its attributes through to the <input>
        # elements, so in order to get the javascript onClick we add it here.
        return [re.sub(r'input ','input onClick="activate_widgets()" ',str(i))
             for i in field.as_widget(field.field.widget)]
 

While editing the rendered content is a bit ooky, it's the only way I've been able to find to add javascript for now.

Custom validation in django newforms

Let's say that we have a form, with some form elements conditionally required. E.g., if you've picked a wg action in mail_type, then you must have picked a working group in group. Here's the form we're working with:

class ListReqStep1(forms.Form):
    DOMAIN_CHOICES = (
        ('ietf.org', 'ietf.org'),
        ('iab.org', 'iab.org'),
        ('irtf.org', 'irtf.org'),
    )
    mail_type = forms.ChoiceField(choices=(
        ('newwg', 'Create new WG email list at ietf.org'),
        ('movewg', 'Move existing WG email list to ietf.org'),
        ('closewg', 'Close existing WG email list at ietf.org'),
        ('newnon', 'Create new non-WG email list at selected domain above'),
        ('movenon', 'Move existing non-WG email list to selected domain above'),
        ('closenon', 'Close existing non-WG email list at selected domain above'),
        ), widget=forms.RadioSelect())
    group = forms.ChoiceField(choices=..., required=False)
    domain_name = forms.ChoiceField(choices=DOMAIN_CHOICES, required=False)
    list_to_close = forms.ChoiceField(choices=..., required=False)

First, the simplest: list_to_close is required if mail_type is "closenon". So we make it required=False in the definition and add some field-specific validation in the field's clean function:

    def clean_list_to_close(self):
        if self.clean_data.get('mail_type', '') == 'closenon':
            if not self.clean_data.get('list_to_close'):
                raise forms.ValidationError, 'Please pick a list to close'
        return self.clean_data['list_to_close']

Note that the clean function has to return the cleaned value. In this case, we're not actually cleaning anything, just making our own 'conditionally required' validation.

More complex validation

When the action is an action for a working group, we first want to make the group list 'conditionally required', as above. However, we also want to check, if request is to close a list, that the list exists, or if the request is to create a list, that the list doesn't exist yet. So we fetch the count from the ImportedMailingList.objects, and depending on the action, we error if there are none or if there aren't none.

 def clean_group(self):
     group = self.clean_data['group']
     action = self.clean_data.get('mail_type', '')
     if action.endswith('wg'):
         if not self.clean_data.get('group'):
             raise forms.ValidationError, 'Please pick a working group'
         group_list_exists = ImportedMailingList.objects.filter(group_acronym=group).count()
         if action.startswith('close'):
             if group_list_exists == 0:
                 raise forms.ValidationError, 'The %s mailing list does not exist.' % group
         else:
             if group_list_exists:
                 raise forms.ValidationError, 'The %s mailing list already exists.' % group
     return self.clean_data['group']
 

Once again, we have to end up returning the data from the clean function. (Note that in the real implementation, group is actually a ModelChoiceField, so using it in the filter provides the numeric ID, and using it in the ValidationError uses the str() representation.)

Since I'm just performing validation, none of my clean_ functions change the data that is returned, but that is quite reasonable - if you have a string that you want to trim leading and trailing spaces from, or do some other kind of normalization on, this is also a hook for that.

---

P.S.

Note that in django SVN, the clean_data attribute has been renamed to cleaned_data, so any references have to be changed accordingly. If you're using any version past 0.96, be aware of this change.

Set the database charset for django inspectdb

I'm starting to use django, which is looking like a really excellent web application framework, to build an application with a legacy database data model. It comes with a really promising tool for this purpose: "inspectdb". You just set your database connection options, run "python manage.py inspectdb", and it outputs a really good start at a data model for your database.

I had two problems:
1. The character fields came out 3 times as large as they were in the database.
2. Some CHAR fields came out as TextField, not CharField.

The solution to #1 is to set the charset properly when talking to the database. This database has lived since 1975 (ok, not really, but it seems like it), so obviously isn't in Unicode. Django 0.96's defaults when talking to the database include charset: 'utf8'. MySQLdb seems to include padding for UTF-8 character expansion. To work around this, I put the following in my settings.py:

 DATABASE_OPTIONS={'charset': 'Latin1',}

#2 looks like an oversight in django's mysql backend - it knows to translate VARCHAR to CharField, but CHAR it allows to default to TextField. A simple patch, submitted to django's trac as #4048, fixes this to behave as I hoped.

Adding new data sources to an RRD using XSLT

The perl module RRD::Simple includes an add_source() function, which mucks with the XML format of an RRD to add data sources. For stupid reasons, CPAN wasn't working on the system that contained the RRDs I wanted to muck with, so instead of fixing that I decided to figure out what RRD::Simple did and do it with XSLT. (OK, so jab pushed me in this direction, I was going to fix CPAN...)

1. Figure out what the XML declaration of the new sources should look like. The easiest way to do this is to create an RRD and then dump it and look at the <ds> elements of the dumped file.

2. Create an xsl file to add these sources. Here's what I did:

<?xml version="1.0"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
  <xsl:template match="@*|node()">
    <xsl:copy>
      <xsl:apply-templates select="@*|node()"/>
    </xsl:copy>
  </xsl:template>
  <xsl:template match="/rrd/ds[position()=last()]">
    <xsl:copy-of select="."/>
    <ds>
...new ds elements go here...
    </ds>
  </xsl:template>
  <xsl:template match="/rrd/rra/cdp_prep/ds[position()=last()]">
    <xsl:copy-of select="."/>
    <ds>
      <primary_value> 0.0000000000e+00 </primary_value>
      <secondary_value> 0.0000000000e+00 </secondary_value>
      <value> NaN </value>
      <unknown_datapoints> 0 </unknown_datapoints>
    </ds>
...one of the above <ds>...</ds> for each new data source...
  </xsl:template>
  <xsl:template match="/rrd/rra/database/row/v[position()=last()]">
    <xsl:copy-of select="."/>
    <v> NaN </v>
...one NaN line for each new data source...
  </xsl:template>
</xsl:stylesheet>

3. Process each rrd with rrdtool dump, xsltproc, rrdtool restore. Here's my shell script.

#!/bin/sh
cd /opt/local/share/cacti/rra
for i in *_yorkie_*.rrd
do
        if [ $i = "bills_yorkie_uptime_26.rrd" ]
        then
                continue
        fi
        x=`basename $i .rrd`.xml
        x2=`basename $i .rrd`-new.xml
        rrdtool dump $i $x
        xsltproc /Users/fenner/src/yorkie/add-ds.xsl $x > $x2
        mv $i $i.old
        rrdtool restore $x2 $i
done

Installing updated timezone info on old FreeBSD

I installed the 2007 timezone info on my old FreeBSD box, so that I didn't have to worry about daylight savings in March. There's a much easier way to do this if you have a system on which ports are still supported, which is to install misc/zoneinfo. If you have an older box like I do, these instructions might help.

Steps:

1. Download timezone data. (Note that as new versions are published, the filename gets updated. Substitute the new filename as appropriate.)

cd /tmp
fetch ftp://elsie.nci.nih.gov/pub/tzdata2007a.tar.gz

2. Extract it

cd /tmp
mkdir tz
cd tz
tar zxvf ../tzdata2007a.tar.gz

3. Run zic

zic -d /usr/share/zoneinfo -p America/New_York  -u root -g wheel
  africa antarctica asia australasia etcetera europe
  factory northamerica southamerica systemv

4. Install zone.tab

install -o root -g wheel -m 444  zone.tab /usr/share/zoneinfo/

5. Fix /etc/localtime (on some systems it's a symlink; on mine it was a copy)

cp /usr/share/zoneinfo/America/Los_Angeles /etc/localtime

6. Test it (note, this step actually changes the clock. I had an unexpected cron job run so be careful!)

date 200703110159; sleep 61; date

Test output should be something like

Sun Mar 11 01:59:00 PST 2007
Sun Mar 11 03:00:01 PDT 2007

7. Set your clock back.

Don't bind to 255.255.255.255

I was trying to debug a problem netbooting a Sun system, as the beginning of a jumpstart process. This process uses RARP and then a tftp of the boot loader from the system that answered the RARP.

1: 07:02:23.142790 rarp who-is 00:03:ba:1a:fa:43 tell 00:03:ba:1a:fa:43
2: 07:02:23.143043 rarp reply 00:03:ba:1a:fa:43 at 172.21.14.15
3: 07:02:23.143446 IP 172.21.14.15.32769 > 255.255.255.255.tftp:  17 RRQ "AC150E0F" octet
4: 07:02:23.144024 IP 172.21.14.17.32806 > 172.21.14.15.32769: UDP, length 516
5: 07:02:23.144540 IP 172.21.14.15.32769 > 172.21.14.17.32806: UDP, length 4
6: 07:02:23.144553 IP 172.21.14.17 > 172.21.14.15: icmp 40:
                      172.21.14.17 udp port 32806 unreachable
7: 07:02:27.162049 IP 172.21.14.15.32769 > 172.21.14.17.32806: UDP, length 4
8: 07:02:27.162073 IP 172.21.14.17 > 172.21.14.15: icmp 40:
                      172.21.14.17 udp port 32806 unreachable

What in the world could be going on here? The server, 172.21.14.17, replied from port 32806 in packet 4, but when the booting client replied in packet 5, the server replied with an error in packet 6, that the port was unreachable.

The key is that the tftp server code uses the address to which the request was sent (packet 3) to bind to. This is important on a multi-homed host or a system with multiple IP addresses for virtual hosting - if it decides to reply from a different IP address, the tftp client may not accept the packet for security reasons.

However, if you bind to 255.255.255.255, apparently Linux will send the packet out with the interface's address, but will fail to demux the incoming packet since it's not addressed to 255.255.255.255.

The tftp server in question (atftp) used the following code sequence:

    bind(s, &to, sizeof(to))
    getsockname(s, &bound, ...)
    connect(s, &client, sizeof(client))

The right thing to do here is to only bind if it's not the broadcast, and move the getsockname after the connect so that if it wasn't explicitly bound, the implicit bind performed by connect still gives it an address and port.

    if (to.sin_addr.s_addr != INADDR_BROADCAST)
        bind(s, &to, sizeof(to))
    connect(s, &client, sizeof(client))
    getsockname(s, &bound, ...)

This is one of the more subtle items in the socket interface. On some systems, the bind would have failed immediately, causing the error to be reported by the tftp server; on others, you get this hard-to-debug behavior.