recursivity elegance

while searching for an elegant solution, I stumbled upon this by Andrew Clark

incoming = {'E': {'D': {'A': {},
                        'O': {'Co': {},
                              'Cu': {}
                             }
                        }
                 }
            }

expected = [
   ('E', 'D', 'A'),
   ('E', 'D', 'O', 'Co'),
   ('E', 'D', 'O', 'Cu'),
]


def paths(tree, cur=()):
    if not tree:  # previous was a leaf, path to us is a solution
        yield cur
    else:  # we're a node
        for node, subtree in tree.iteritems():
            for path in paths(subtree, cur+(node,)):
                yield path

assert list(paths(incoming)) == expected

Generate unit tests for sqlalchemy mappings

When playing with sqlalchemy mappings, it seems a good idea do unit test them, just to make sure they compile properly. So, given this simple mappings (from sqla tutorial)

__author__ = 'nlaurance'
from sqlalchemy import Column, DateTime, String, Integer, ForeignKey, func
from sqlalchemy.orm import relationship, backref
from sqlalchemy.ext.declarative import declarative_base


Base = declarative_base()


class Department(Base):
    __tablename__ = 'department'
    id = Column(Integer, primary_key=True)
    name = Column(String, nullable=False)


class Employee(Base):
    __tablename__ = 'employee'
    id = Column(Integer, primary_key=True)
    name = Column(String)
    hired_on = Column(DateTime, default=func.now())
    department_id = Column(Integer, ForeignKey('department.id'))
    department = relationship(
        Department,
        backref=backref('employees',
                         uselist=True,
                         cascade='delete,all'))

it would make sense to, at least make sure we can create one and retrieve its instance Let's create a test base, that our tests will inherit from, and factor out sensible defaults that could be used in integration tests later on.

__author__ = 'nlaurance'
import inspect
from unittest import TestCase
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqla_classes import Base
from sqla_classes import Department, Employee


DEFAULTS = {
    Department: {
        "name": "Lorem",
    },
    Employee: {
        "name": "Lorem",
    },
}


def create_some(session, klass, values=None):
    default = DEFAULTS.get(klass, {})
    values = values or [default]
    for value in values:
        new_default = default.copy()
        new_default.update(value)
        session.add(klass(**new_default))


class SqlAMappingsBase(object):

    engine = create_engine('sqlite:///:memory:')
    session = sessionmaker(bind=engine)()

    class_under_test = None
    default_values = {}

    def setUp(self):
        Base.metadata.create_all(self.engine)
        create_some(self.session, self.class_under_test)
        self.session.commit()

    def tearDown(self):
        Base.metadata.drop_all(self.engine)

    def test_get(self):
        self.session.query(self.class_under_test).one()

now if we want to unit test Department and Employee classes we would write something like :

class DepartmentTest(SqlAMappingsBase, TestCase):
    class_under_test = Department
    default_values = DEFAULTS.get(Department, {})

class EmployeeTest(SqlAMappingsBase, TestCase):
    class_under_test = Employee
    default_values = DEFAULTS.get(Employee, {})

so far, so good. but ... what if the projects requires to create lots of mappings, we would end up with lots of look alike code, mostly copied/pasted from one of the examples. Why not create the test classes on the fly ? some code like ..

def autotest_mappings(module_under_test, test_module):
    for name, klass in inspect.getmembers(module_under_test, inspect.isclass):
        if Base in inspect.getmro(klass) and klass is not Base:
            test_name = ''.join((name, 'Test'))
            test_klass = type(test_name, (SqlAMappingsBase, TestCase),
                              {"class_under_test": klass,
                               "default_values": DEFAULTS.get(klass, {})})
            setattr(test_module, test_name, test_klass)

this function takes as arguments the module we want to test, and the test module in which to insert the tests. It then looks for all classes that inherit from Base, create a new class and inject it into the test module. Our test code would then look like :

from our_project import models
from sqla_utils import autotest_mappings

autotest_mappings(models, sys.modules[__name__])

and that's it ! three lines and no matter how many classes you declare in your models module, each would be picked by the inspect sniffer and would have a nice unit test.

My setup of internal network with virtual boxes

My goal today, was to setup 4 virtual boxes that can communicate over an internal virtual network, cloning a previously configurated vbox.

I mostly used the GUI.
The network config is as such:

 Note: the bridged interface (so that those vboxes have internet access) has its "Promiscuous Mode" set to "Allow VMs"

Time to clone

Note: Check the Mac reinitialize


Now boot the clone, you'll notice it waits for network configuration.

log in and set /etc/network/interfaces

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet dhcp

auto eth1
iface eth1 inet static
 address 10.0.0.11
 network 10.0.0.0
 netmask 255.255.255.0 

ifconfig -a will show eth3 eth4 .. not want we want

each time you give a new MAC address (in VBox guest settings) Debian and Ubuntu guests udev assigns a new eth number
edit /etc/udev/rules.d/70-persistent-net.rules or delete the file: it will be recreated @ next boot

ubuntu@ubuntu:~$ sudo rm /etc/udev/rules.d/70-persistent-net.rules

and reboot.

check interfaces

ubuntu@ubuntu:~$ ifconfig
eth0      Link encap:Ethernet  HWaddr 08:00:27:9a:a2:5a  
          inet addr:192.168.1.73  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe9a:a25a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:241 errors:0 dropped:0 overruns:0 frame:0
          TX packets:156 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:30767 (30.7 KB)  TX bytes:22818 (22.8 KB)

eth1      Link encap:Ethernet  HWaddr 08:00:27:d3:7d:59  
          inet addr:10.0.0.11  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fed3:7d59/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:468 (468.0 B)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:16 errors:0 dropped:0 overruns:0 frame:0
          TX packets:16 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:1184 (1.1 KB)  TX bytes:1184 (1.1 KB)

check physical network dhcp handled the bridged interface

ubuntu@ubuntu:~$ ping google.com
PING google.com (173.194.41.66) 56(84) bytes of data.
64 bytes from lhr08s01-in-f2.1e100.net (173.194.41.66): icmp_req=1 ttl=51 time=39.1 ms

boot another "clone", proceed as aforementioned and check

ubuntu@ubuntu:~$ ping 10.0.0.11
PING 10.0.0.11 (10.0.0.11) 56(84) bytes of data.
64 bytes from 10.0.0.11: icmp_req=1 ttl=64 time=0.049 ms 

http://www.virtualbox.org/manual/ch06.html

Install Postgis with ansible

My goal was to automate the installation of postgres and its postgis extension on a virtual machine, following more or less this procedure.
I was talked into trying ansible and decided to head this way.


At first I installed a vanilla Ubuntu12.04LTS on a virtual box. I've just added my ssh public key.

Then I followed the general instruction to have ansible running.


I came up with two playbooks, one to install the RDMS and its dependencies, the other to add a postgis enabled database added.

my hosts file is like :

[vms]
vm1204 ansible_ssh_host=192.168.1.64 userhome=/home/remoteuser

Install Postgres + PostGIS

---

- hosts: vms
  sudo: True
  gather_facts: False

  tasks:
  - name: ensure apt cache is up to date
    action: apt update_cache=yes
  - name: ensure packages are installed
    action: apt pkg=$item
    with_items:
        - build-essential
        - postgresql-9.1
        - postgresql-server-dev-9.1
        - libxml2-dev
        - proj
        - libjson0-dev
        - xsltproc
        - docbook-xsl
        - docbook-mathml
        - libgdal1-dev


- hosts: vms

  tasks:
  - name: create download dir
    action: file dest=${userhome}/download state=directory
    
  - name: download GEOS
    action: get_url url=http://download.osgeo.org/geos/geos-3.3.5.tar.bz2 
                    dest=${userhome}/download/geos-3.3.5.tar.bz2 mode=0440
  - name: untar GEOS
    action: command tar xjf geos-3.3.5.tar.bz2 chdir=${userhome}/download/
  - name: configure GEOS
    action: command ./configure chdir=${userhome}/download/geos-3.3.5
  - name: make GEOS
    action: command make chdir=${userhome}/download/geos-3.3.5
    
    
- hosts: vms
  sudo: True    
  
  tasks:
  - name: install GEOS
    action: command make install chdir=${userhome}/download/geos-3.3.5
    
    
- hosts: vms

  tasks:
  - name: create download dir
    action: file dest=${userhome}/download owner=isisafe state=directory
        
  - name: download PostGis
    action: get_url url=http://postgis.org/download/postgis-2.0.1.tar.gz 
                    dest=${userhome}/download/postgis-2.0.1.tar.gz mode=0440
  - name: untar PostGis
    action: command tar xzf postgis-2.0.1.tar.gz chdir=${userhome}/download/
  - name: make PostGis
    action: command make chdir=${userhome}/download/postgis-2.0.1
    
    
- hosts: vms
  sudo: True    
  
  tasks:
  - name: install PostGis
    action: command make install chdir=${userhome}/download/postgis-2.0.1
  - name: install PostGis
    action: command make comments-install chdir=${userhome}/download/postgis-2.0.1
  - name: post install ldconfig
    action: command ldconfig
   

then to launch it:

nil@home$ ansible-playbook create_postgis_db.yaml -u remoteuser -K -v

sudo password: 

The install took a very long time on the VM, I should get a better computer.

Creating a PostGIS enabled database

---

- hosts: vms
  sudo: True
  sudo_user: postgres
  gather_facts: False

  vars_prompt:
    - name: "dbuser"
      prompt: "user login"
      private: False    
    - name: "dbname"
      prompt: "database name"
      private: False       
    - name: "dbpassword"
      prompt: "user pwd"
      private: True    
    
  tasks:
      - name: ensure database is created
        action: postgresql_db db=$dbname
        notify:
          - convert database to postgis
          - convert database to postgis topology
    
      - name: ensure user has access to database
        action: postgresql_user db=$dbname user=$dbuser password=$dbpassword priv=ALL

      - name: ensure user does not have unnecessary privilege
        action: postgresql_user user=$dbuser role_attr_flags=NOSUPERUSER,NOCREATEDB

  handlers:
      - name: convert database to postgis
        action: command psql -d $dbname -c "CREATE EXTENSION postgis;"
      - name: convert database to postgis topology
        action: command psql -d $dbname -c "CREATE EXTENSION postgis_topology;"


    

Let's add a new database on the remote server

nil@home$ ansible-playbook create_postgis_db.yaml -u remoteuser -K -v
sudo password: 
user login: nil
database name: new_map
user pwd:

check on the VM that the database was created

postgres@vm1204$ psql 

postgres=# \l
                                  List of databases
    Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges   
------------+----------+----------+-------------+-------------+-----------------------
 new_map    | postgres | UTF8     | fr_FR.UTF-8 | fr_FR.UTF-8 | =Tc/postgres         +
            |          |          |             |             | postgres=CTc/postgres+
            |          |          |             |             | nil=CTc/postgres

postgres@vm1204$ psql new_map

new_map=# \dt
               List of relations
  Schema  |      Name       | Type  |  Owner   
----------+-----------------+-------+----------
 public   | spatial_ref_sys | table | postgres
 topology | layer           | table | postgres
 topology | topology        | table | postgres

Note : I don't clean up the source files after PG install

Customize django_tinymce widget in django admin

The lazy path using an HTMLField can be customized easily:

in settings.py

TINYMCE_DEFAULT_CONFIG = {
    'plugins': "xhtmlxtras",
    'theme': "advanced",
    "theme_advanced_buttons1" : "bold,italic,underline,separator,bullist,numlist,separator,outdent,indent,separator,undo,redo",
    "theme_advanced_buttons2" : "link,unlink,separator,removeformat,separator,sub,sup,separator,abbr",
    "theme_advanced_buttons3" : "",
    'cleanup_on_startup': True,
    'custom_undo_redo_levels': 10,
}
 

all options :

• http://www.tinymce.com/wiki.php/Configuration:Advanced_theme
• http://www.tinymce.com/wiki.php/Buttons/controls

change the size of string column using sqlalchemy migrate

Why on earth have I done a table description like

CREATE TABLE address
(
  id serial NOT NULL,
  street text,
  city character varying(255),
  province character varying(10),
...

what a scrooge, 10 chars only, you can't live in saskatchewan with that. I had to migrate.

After a while I managed to get the syntax that makes it work

def upgrade(migrate_engine):
    meta.bind = migrate_engine
    address = Table('address', meta, autoload=True)
    address.c.province.alter(type=VARCHAR(length=255))

Eclipse pydev template to generate sql alchemy migrations

How often have you committed code, just to realize your missing a column in your sqlAlchemy model?
If you're using eclipse IDE, you can leverage the templates to write a migration script in no time.
Go to windows -> preferences -> PyDev -> Editor -> Templates

then create a new template with :

• name : Module: Migration SQLA
• context: New Module
• description: SQLA migration

# -*- coding: utf-8 -*-
'''
Created on ${date}

@author: ${user}
'''

from sqlalchemy import *
from migrate import *

meta = MetaData()

def upgrade(migrate_engine):
    meta.bind = migrate_engine
    some = Table('some',  meta, autoload=True)
    if not 'somename' in some.columns:
        new_col = Column('somename', Unicode(255))
        new_col.create(some)
            ${cursor}

def downgrade(migrate_engine):
    meta.bind = migrate_engine
        some = Table('some',  meta, autoload=True)
        some.c.some_col.drop()

And that's it ! you can now go into your migrations folder and do:
New -> PyDev Module, and choose your new template