Save the ions!

Modifying the regexp for usernames in django-admin

About a month ago, I built a centralized authentication backend using django-roa for my company, AUF. It’s actually a cascading authentication system, since I copy locally (in Django’s own authentication system) the account, once validated.

That was working just fine, until I tried adding some of my users’ local accounts in different groups. The distant usernames follow the format <first_name>.<last_name>, and since Django doesn’t allow that, everytime I would try to save the user edit form in the admin interface, I would get an error message saying that the username format was invalid.

After looking around a bit, I found this solution, which has been working great for me.

from django.contrib.auth.forms import UserCreationForm, UserChangeForm
from django.contrib.auth.models import User
from django.contrib.auth.admin import UserAdmin
from django import forms
from django.utils.translation import ugettext_lazy as _
from django.contrib import admin

class UserCreationForm(UserCreationForm):
    username = forms.RegexField(label=_("Username"), max_length=30,
            regex=r"^[\w'\.\-]+\s?[\w'\.\-]+$")

class UserChangeForm(UserChangeForm):
    username = forms.RegexField(label=_("Username"), max_length=30,
            regex=r"^[\w'\.\-]+\s?[\w'\.\-]+$")

class UserProfileAdmin(UserAdmin):
    form = UserChangeForm
    add_form = UserCreationForm

admin.site.unregister(User)
admin.site.register(User, UserProfileAdmin)

UVa

A while back, when I was a student in Paris, one of our teachers told us about some project from University of Valladolid (I can’t remember the actual name of the project), it was a contest for programmers, and the goal was to solve as many simple algorithmic problems as possible.

At that time, I thought it was pretty cool, since most of the class was only beginning to understand programming (the class was something like C programming 101), so I used to look at problems and solved them.

I was thinking about that a couple of weeks ago, and I found out that the project still exists, it only changed location/name, which means that I lost my account data (it’s not that bad), but which also means I can keep having fun with it!

You know, by having fun, I mean actually having to think of algorithms and write them myself, you can read this article if you’re not sure what I mean.

Anyway, long story short, the project is UVa Online Judge and it’s awesome!

As a bonus, here’s my solution for the first problem: The 3n + 1 problem

Have fun!

 

// Q100 - The 3n + 1 problem
// http://bit.ly/2aRM6Y
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <sys/types.h>

#define MAXNUMBER 1000000

unsigned int cycleLength (unsigned int *known, unsigned int i)
{
  unsigned int length;

  if (i == 1)
    length = 1;
  else if (i < MAXNUMBER && known[i] != 0)
    length = known[i];
  else if (i % 2)
    {
      length = 1 + cycleLength (known, i * 3 + 1);
      if (i < MAXNUMBER)
        known[i] = length;
    }
  else
    {
      length = 1 + cycleLength (known, i / 2);
      if (i < MAXNUMBER)
        known[i] = length;
    }

  return length;
}

int main (int argc, char **argv)
{
  unsigned int *knownCycles;
  unsigned int max, len, a, b, i, start, end;

  knownCycles = calloc (MAXNUMBER, sizeof (unsigned int));

  while (scanf ("%u %u", &a, &b) != EOF)
    {
      max   = 0;
      start = (a > b) ? b : a;
      end   = (a > b) ? a : b;
      for (i = start; i <= end; i++)
        {
          len = cycleLength (knownCycles, i);
          if (len > max)
            max = len;
        }
      printf ("%u %u %u\n", a, b, max);
    }

  free (knownCycles);

  return 0;
}

Fun VoIP use

As I was cleaning up data on my cellphone tonight, I found a video I had recorded about a year ago. I was working -as a side-contract- on a VoIP telephony / PA / bell system for schools. At the lab I was working, I had about 10 phones, 6 speakers (for the hallways), and an intercom (can’t remember why). I was testing the bell system, which was supposed to play a sound on all those devices, from a shell script (actually python, IIRC) that would connect to the asterisk server. The first “beep” is made by all the phones, it’s an alert saying an announce is coming :)
I record one of those tests, and even though it was recorded by my crappy but oh-so-reliable BenQ phone. The music playing is Radiohead’s All I Need.

A little propaganda!

I’ll ba attending those 2 events in the next couple of weeks!

I will be at GeekFest Montreal next March 6-7

 

confoo.ca Web Techno Conference

Introducing django-exportateur

Exportateur?

OK, this article is a bit late, but I think it’s still necessary to properly introduce a new project.

At work last week, I had to write a generic way of exporting a QuerySet to csv, ods, and maybe some other format later on. Since I work at AUF, I frenched up the project’s name.

Why start from scratch?

I’m pretty sure that there already are a few apps that do that, but since I still consider myself kind of new to django, I thought it would be a good learning exercise to write this app myself.

The idea

The files I had to generate needed to have a header that described each column, and the values needed to be clear (no ids, etc.)

I decided to add a method to get the object’s fields (attribute name as well as verbose name), and another one to get the attributes’ values. That’s pretty much it for the Django part.

The last part of the project was to define a generic-enough method to export those fields in the file format we want. The file format is passed as a parameter so that it can be passed directly from the url.

Have a look

The code is now published on code.google.com. Feedback is always welcome.

Logging your instances' history in Django

History tables?

While working on a rewrite of one of our web app here at AUF, I needed to replicate a logging functionality.

The old system was written with php4 (please don’t barf on your screen), and so you can imagine how such a thing could have been done:

  1. Get current entry in the table.
  2. Generate an insert query to save that version of the entry in the history table.
  3. Perform the update on the main table.

Being lazy, as all programmers are, I looked around for an existing django app to do that, and I found django-history-tables, which does exactly what I need!

Long story short, the latest release was for django 0.96, which is pretty old, and so I had to modify it a bit to make it work on 1.1.1, and later on, I found out that it was really unstable on slightly complex models. Please note that this is by no means criticism towards django-history-tables’ author. I failed at fixing it.

So, what happens next? When the only solution I have fails, I have to buid one! Yay!

The concept

I figured the easiest way to this would require a couple of things:

  • The model for the history table must be created by a syncdb
  • It must be relatively easy to use

I decided then to start writing an abstract History, whose only job is to define a generic-enough saving method.

Here’s what it looks like

#!/usr/bin/env python
# -*- coding: utf-8 -*-
from django.db import models
from django.db.models.base import ModelBase
from django.db import connection, transaction
from copy import deepcopy
from datetime import datetime

class History (models.Model):
    history_datetime = models.DateTimeField(default=datetime.now)
    history_objectid = models.PositiveIntegerField()
    history_revision = models.PositiveIntegerField()
    history_comment = models.CharField(max_length=1024, default="")

    def save_log (self, instance):
        iClass = instance.__class__
        hClass = self.__class__

        # Reload the instance from db
        instance = iClass.objects.get(id = instance.id)
        self.history_objectid = instance.id
        self.history_comment = "pre_save history item <%s>" \
                % (repr(instance))
        self.history_revision = \
            hClass.objects.filter(history_objectid=instance.id).count()+1

        for field in instance._meta.fields:
            if field.__class__.__name__ != 'AutoField':
                if getattr(field, 'unique', False):
                    field._unique = False
                setattr(self, field.name, getattr(instance, field.name))

        super (History, self).save ()

    def save (self):
        pass

    class Meta:
        abstract = True

This class is pretty straightforward, the only odd thing is with the save method. I had to do this little hack because the rows in the history table would end up duplicated.

Usage

I think the best way to explain usage is by showing an example:

from django.db import models
from django_historique import History
from django.db.models.signals import pre_save, pre_delete

class Brand (models.Model):
    nom = models.CharField (max_length = 32)

    def __unicode__ (self):
        return "<Brand %s>" % self.nom

class CarBase (models.Model):
    marque = models.ForeignKey (Brand)
    couleur = models.CharField (max_length = 32)

    class Meta:
        abstract = True

class CarHistory (History, CarBase):
    def __unicode__ (self):
        return "<CarH %s, rev. %s>" \
                % (self.history_objectid, self.history_revision)

class Car (CarBase):
    id = models.AutoField (primary_key = True)
    def __unicode__ (self):
        return "<Car %s>" % self.id

def carhistory_save (sender, instance, signal, *args, **kwargs):
    history = CarHistory ()
    history.save_log (instance)
pre_save.connect(carhistory_save, sender=Car)
pre_delete.connect(carhistory_save, sender=Car)

A few comments:

  • I do need to define an abstract CarBase that doesn’t include the id first, otherwise, django’s ORM screams that it’s not unique in the aggregate CarHistory class.
  • The pre_save and pre_delete are what django offers us to execute actions before saving and deleting instances.

Issues

I can save ForeignKey values, so you will need to add a *History class for your foreign items and work some kind of time-based logic to figure out which version of the foreign object your instance was referring to

I don’t support ManyToMany fields

I hope you enjoyed it, the source code will be published on monday at google code: django-historique

Packaging python applications/modules for Debian

I got to work on a fun project yesterday, I needed to build a script that could generate a Debian package from a python module.

Yes, that does sound easy enough… Except that it’s not!

First of all, how does one build a .deb package? There’s lots of documentation on that, probably way too much, even. The main idea is that you will have to create a debian folder and add a few files in there that will define your package and how to build it, and then by running debuild, it will run a somewhat typical ./configure; make clean; make; make install

Issues

One of the problems with that is that you generally don’t have nor want to write a Makefile for your Python project… I mean, what would it even do?

The other issue I had was that Debian doesn’t want us to install just eggs through a deb package, which is understandable, you don’t want to install a package of a package.

Let’s get started!

I will assume you’re working on a project named ninja, and that it has this structure:

ninja/
  core/
    ninjalib.py
    something.py
  ninja.py
  readme.txt

Now, the first things you’ll need to do is to create (directly in the ninja folder) the Makefile and the debian folder, which will contain at least the following files:

Makefile
debian/
  changelog
  control
  dirs
  docs
  postinst
  rules

Each of these files needs to follow a strict format, and contains very specific information:

Makefile

Since python doesn’t require us to actually build anything, we’re going to use the Makefile only to put our files in place.

Example

LIBDIR = $(DESTDIR)/usr/share/pyshared/ninja
BINDIR = $(DESTDIR)/usr/bin
clean:
        rm -f *.py[co] */*.py[co]
install:
        mkdir -p $(LIBDIR)
        mkdir -p $(BINDIR)
        cp -r core $(LIBDIR)/
        cp ninja.py $(BINDIR)/ninja
uninstall:
        rm -rf $(LIBDIR)
        rm -f $(BINDIR)/ninja

A few things to note here:

  • $(DESTDIR) is an environment variable, defined in your rules file, which represents the root of the build environment (think chroot).
  • clean removes the dirty python bytecode.
  • install should put in place all required code, and uninstall should remove it.
  • Makefiles allow only tabs, be careful.

changelog

This file should be your project’s changelog, but it must comply with the debian format, see example:

ninja (0.1) unstable; urgency=low
    * Initial version
 -- Cyril Ninja <ninja@...>  Wed, 20 Jan 2010 18:47:11 -0500

The entire file should consist of these blocks, but be careful:

  • (0.1) is the version number for the package you’re building. debuild will always use the top-most version number.
  • You should have as many lines starting with * as there are changes in this version.
  • It is an actual tab before the *, no spaces are allowed.
  • The last line starts with a space, and there are 2 spaces between the email and the date.

control

In this file, you have to define what your packages actually are, and what they depend on (also what other packages they suggest, if you wish to do that).

Let’s see an example:

Source: ninja
Section: python
Priority: extra
Maintainer: Cyril Ninja 
Build-Depends: debhelper (>= 7), python-support
Standards-Version: 3.8.0
Homepage: http://ninja

Package: ninja
Architecture: all
Depends: python (>= 2.6), python-ninja-foundation
Description: This is powerful ninja stuff

This example defines two packages: a source one, and a binary one. This is pretty much straightforward, so I’ll leave it like that.

dirs

In the Makefile earlier, we defined a couple of directories (LIBDIR, BINDIR) where our code will be installed, and this file should contain the directory names we want to include in our package.

For this project, it should look like this:

usr/bin/
usr/share/pyshared/ninja

docs

You should define in this file all documentation that should be included in the package.

Let’s have a look at the awesome ninja docs file:

readme.txt

It will be installed in /usr/share/doc/ninja

postinst

The postinst file is used to run commands when the package is installed / upgraded / removed. In our case, we only need to add a symlink so that python can find our module.

Example file

#!/bin/sh
set -e
case "$1" in
    configure)
    ln -s /usr/share/pyshared/ninja /usr/lib/python2.6/dist-packages/
    ;;

    abort-upgrade|abort-remove|abort-deconfigure)
    rm -f /usr/lib/python2.6/dist-packages/ninja
    ;;

    *)
        echo "postinst called with unknown argument \`$1'" >&2
        exit 1
    ;;
esac
exit 0

Feel free to add any command you wish in that file.

rules

If you’re anything like me, you’re going to hate that file… Fortunately for us, there is virtually nothing to change from the example below:

#!/usr/bin/make -f
# -*- makefile -*-
configure: configure-stamp
configure-stamp:
        dh_testdir
        touch configure-stamp
build: build-stamp
build-stamp: configure-stamp
        dh_testdir
        $(MAKE)
        touch $@
clean:
        dh_testdir
        dh_testroot
        rm -f build-stamp configure-stamp
        dh_clean
install: build
        dh_testdir
        dh_testroot
        dh_prep
        dh_installdirs
        $(MAKE) DESTDIR=$(CURDIR)/debian/ninja install
binary-arch: build install
        # emptyness
binary-indep: build install
        dh_testdir
        dh_testroot
        dh_installchangelogs
        dh_installdocs
        dh_installexamples
        dh_pysupport
        dh_link
        dh_fixperms
        dh_installdeb
        dh_shlibdeps
        dh_gencontrol
        dh_md5sums
        dh_builddeb
binary: binary-indep
.PHONY: build clean binary-indep binary install configure

  • Simply replace ninja with your project name, and you should be all set.
  • Same here: tabs only.

Ready?

Now that everything seems to be in place, it’s time to build our magnificent Debian package!

From your ninja folder, run

debuild

At some point, if you do have a gpg key mathing the author email address you defined, it should ask for your password twice, to sign both packages.

When it stops, look in the parent folder, you should see

ninja_0.1_all.deb
ninja_0.1.orig.tar.gz
ninja_0.1.dsc

After that, I hope you know what you have to do ;)

Quickly using OpenSSL in C

A little while ago, I was working on a client/server communication module, and I wanted it to be secure. Looking at the documentation I could find, I quickly figured out that it wouldn’t necessarily be easy to do.

Amongst the first issues are the validity of the server’s certificate, which, as I didn’t want to battle with this, I decided to skip

The example below shows how to connect and send/receive data on an SSL-encrypted socket

#include <sys/socket.h>
#include <sys/types.h>
#include <netinet/in.h>
#include <netdb.h>
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <unistd.h>
#include <errno.h>

#include <openssl/rand.h>
#include <openssl/ssl.h>
#include <openssl/err.h>

// Simple structure to keep track of the handle, and
// of what needs to be freed later.
typedef struct {
    int socket;
    SSL *sslHandle;
    SSL_CTX *sslContext;
} connection;

// For this example, we'll be testing on openssl.org
#define SERVER  "www.openssl.org"
#define PORT 443

// Establish a regular tcp connection
int tcpConnect ()
{
  int error, handle;
  struct hostent *host;
  struct sockaddr_in server;

  host = gethostbyname (SERVER);
  handle = socket (AF_INET, SOCK_STREAM, 0);
  if (handle == -1)
    {
      perror ("Socket");
      handle = 0;
    }
  else
    {
      server.sin_family = AF_INET;
      server.sin_port = htons (PORT);
      server.sin_addr = *((struct in_addr *) host->h_addr);
      bzero (&(server.sin_zero), 8);

      error = connect (handle, (struct sockaddr *) &server,
                       sizeof (struct sockaddr));
      if (error == -1)
        {
          perror ("Connect");
          handle = 0;
        }
    }

  return handle;
}

// Establish a connection using an SSL layer
connection *sslConnect (void)
{
  connection *c;

  c = malloc (sizeof (connection));
  c->sslHandle = NULL;
  c->sslContext = NULL;

  c->socket = tcpConnect ();
  if (c->socket)
    {
      // Register the error strings for libcrypto & libssl
      SSL_load_error_strings ();
      // Register the available ciphers and digests
      SSL_library_init ();

      // New context saying we are a client, and using SSL 2 or 3
      c->sslContext = SSL_CTX_new (SSLv23_client_method ());
      if (c->sslContext == NULL)
        ERR_print_errors_fp (stderr);

      // Create an SSL struct for the connection
      c->sslHandle = SSL_new (c->sslContext);
      if (c->sslHandle == NULL)
        ERR_print_errors_fp (stderr);

      // Connect the SSL struct to our connection
      if (!SSL_set_fd (c->sslHandle, c->socket))
        ERR_print_errors_fp (stderr);

      // Initiate SSL handshake
      if (SSL_connect (c->sslHandle) != 1)
        ERR_print_errors_fp (stderr);
    }
  else
    {
      perror ("Connect failed");
    }

  return c;
}

// Disconnect & free connection struct
void sslDisconnect (connection *c)
{
  if (c->socket)
    close (c->socket);
  if (c->sslHandle)
    {
      SSL_shutdown (c->sslHandle);
      SSL_free (c->sslHandle);
    }
  if (c->sslContext)
    SSL_CTX_free (c->sslContext);

  free (c);
}

// Read all available text from the connection
char *sslRead (connection *c)
{
  const int readSize = 1024;
  char *rc = NULL;
  int received, count = 0;
  char buffer[1024];

  if (c)
    {
      while (1)
        {
          if (!rc)
            rc = malloc (readSize * sizeof (char) + 1);
          else
            rc = realloc (rc, (count + 1) *
                          readSize * sizeof (char) + 1);

          received = SSL_read (c->sslHandle, buffer, readSize);
          buffer[received] = '\0';

          if (received > 0)
            strcat (rc, buffer);

          if (received < readSize)
            break;
          count++;
        }
    }

  return rc;
}

// Write text to the connection
void sslWrite (connection *c, char *text)
{
  if (c)
    SSL_write (c->sslHandle, text, strlen (text));
}

// Very basic main: we send GET / and print the response.
int main (int argc, char **argv)
{
  connection *c;
  char *response;

  c = sslConnect ();

  sslWrite (c, "GET /\r\n\r\n");
  response = sslRead (c);

  printf ("%s\n", response);

  sslDisconnect (c);
  free (response);

  return 0;
}

To build this example program using gcc:

gcc -Wall -lssl -lcrypto -o ssl-demo ssl-demo.c

Have fun with SSL!