My setup of internal network with virtual boxes

My goal today, was to setup 4 virtual boxes that can communicate over an internal virtual network, cloning a previously configurated vbox.

I mostly used the GUI.
The network config is as such:

 Note: the bridged interface (so that those vboxes have internet access) has its "Promiscuous Mode" set to "Allow VMs"

Time to clone

Note: Check the Mac reinitialize


Now boot the clone, you'll notice it waits for network configuration.

log in and set /etc/network/interfaces

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet dhcp

auto eth1
iface eth1 inet static
 address 10.0.0.11
 network 10.0.0.0
 netmask 255.255.255.0 

ifconfig -a will show eth3 eth4 .. not want we want

each time you give a new MAC address (in VBox guest settings) Debian and Ubuntu guests udev assigns a new eth number
edit /etc/udev/rules.d/70-persistent-net.rules or delete the file: it will be recreated @ next boot

ubuntu@ubuntu:~$ sudo rm /etc/udev/rules.d/70-persistent-net.rules

and reboot.

check interfaces

ubuntu@ubuntu:~$ ifconfig
eth0      Link encap:Ethernet  HWaddr 08:00:27:9a:a2:5a  
          inet addr:192.168.1.73  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe9a:a25a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:241 errors:0 dropped:0 overruns:0 frame:0
          TX packets:156 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:30767 (30.7 KB)  TX bytes:22818 (22.8 KB)

eth1      Link encap:Ethernet  HWaddr 08:00:27:d3:7d:59  
          inet addr:10.0.0.11  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fed3:7d59/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:468 (468.0 B)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:16 errors:0 dropped:0 overruns:0 frame:0
          TX packets:16 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:1184 (1.1 KB)  TX bytes:1184 (1.1 KB)

check physical network dhcp handled the bridged interface

ubuntu@ubuntu:~$ ping google.com
PING google.com (173.194.41.66) 56(84) bytes of data.
64 bytes from lhr08s01-in-f2.1e100.net (173.194.41.66): icmp_req=1 ttl=51 time=39.1 ms

boot another "clone", proceed as aforementioned and check

ubuntu@ubuntu:~$ ping 10.0.0.11
PING 10.0.0.11 (10.0.0.11) 56(84) bytes of data.
64 bytes from 10.0.0.11: icmp_req=1 ttl=64 time=0.049 ms 

http://www.virtualbox.org/manual/ch06.html

Install Postgis with ansible

My goal was to automate the installation of postgres and its postgis extension on a virtual machine, following more or less this procedure.
I was talked into trying ansible and decided to head this way.


At first I installed a vanilla Ubuntu12.04LTS on a virtual box. I've just added my ssh public key.

Then I followed the general instruction to have ansible running.


I came up with two playbooks, one to install the RDMS and its dependencies, the other to add a postgis enabled database added.

my hosts file is like :

[vms]
vm1204 ansible_ssh_host=192.168.1.64 userhome=/home/remoteuser

Install Postgres + PostGIS

---

- hosts: vms
  sudo: True
  gather_facts: False

  tasks:
  - name: ensure apt cache is up to date
    action: apt update_cache=yes
  - name: ensure packages are installed
    action: apt pkg=$item
    with_items:
        - build-essential
        - postgresql-9.1
        - postgresql-server-dev-9.1
        - libxml2-dev
        - proj
        - libjson0-dev
        - xsltproc
        - docbook-xsl
        - docbook-mathml
        - libgdal1-dev


- hosts: vms

  tasks:
  - name: create download dir
    action: file dest=${userhome}/download state=directory
    
  - name: download GEOS
    action: get_url url=http://download.osgeo.org/geos/geos-3.3.5.tar.bz2 
                    dest=${userhome}/download/geos-3.3.5.tar.bz2 mode=0440
  - name: untar GEOS
    action: command tar xjf geos-3.3.5.tar.bz2 chdir=${userhome}/download/
  - name: configure GEOS
    action: command ./configure chdir=${userhome}/download/geos-3.3.5
  - name: make GEOS
    action: command make chdir=${userhome}/download/geos-3.3.5
    
    
- hosts: vms
  sudo: True    
  
  tasks:
  - name: install GEOS
    action: command make install chdir=${userhome}/download/geos-3.3.5
    
    
- hosts: vms

  tasks:
  - name: create download dir
    action: file dest=${userhome}/download owner=isisafe state=directory
        
  - name: download PostGis
    action: get_url url=http://postgis.org/download/postgis-2.0.1.tar.gz 
                    dest=${userhome}/download/postgis-2.0.1.tar.gz mode=0440
  - name: untar PostGis
    action: command tar xzf postgis-2.0.1.tar.gz chdir=${userhome}/download/
  - name: make PostGis
    action: command make chdir=${userhome}/download/postgis-2.0.1
    
    
- hosts: vms
  sudo: True    
  
  tasks:
  - name: install PostGis
    action: command make install chdir=${userhome}/download/postgis-2.0.1
  - name: install PostGis
    action: command make comments-install chdir=${userhome}/download/postgis-2.0.1
  - name: post install ldconfig
    action: command ldconfig
   

then to launch it:

nil@home$ ansible-playbook create_postgis_db.yaml -u remoteuser -K -v

sudo password: 

The install took a very long time on the VM, I should get a better computer.

Creating a PostGIS enabled database

---

- hosts: vms
  sudo: True
  sudo_user: postgres
  gather_facts: False

  vars_prompt:
    - name: "dbuser"
      prompt: "user login"
      private: False    
    - name: "dbname"
      prompt: "database name"
      private: False       
    - name: "dbpassword"
      prompt: "user pwd"
      private: True    
    
  tasks:
      - name: ensure database is created
        action: postgresql_db db=$dbname
        notify:
          - convert database to postgis
          - convert database to postgis topology
    
      - name: ensure user has access to database
        action: postgresql_user db=$dbname user=$dbuser password=$dbpassword priv=ALL

      - name: ensure user does not have unnecessary privilege
        action: postgresql_user user=$dbuser role_attr_flags=NOSUPERUSER,NOCREATEDB

  handlers:
      - name: convert database to postgis
        action: command psql -d $dbname -c "CREATE EXTENSION postgis;"
      - name: convert database to postgis topology
        action: command psql -d $dbname -c "CREATE EXTENSION postgis_topology;"


    

Let's add a new database on the remote server

nil@home$ ansible-playbook create_postgis_db.yaml -u remoteuser -K -v
sudo password: 
user login: nil
database name: new_map
user pwd:

check on the VM that the database was created

postgres@vm1204$ psql 

postgres=# \l
                                  List of databases
    Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges   
------------+----------+----------+-------------+-------------+-----------------------
 new_map    | postgres | UTF8     | fr_FR.UTF-8 | fr_FR.UTF-8 | =Tc/postgres         +
            |          |          |             |             | postgres=CTc/postgres+
            |          |          |             |             | nil=CTc/postgres

postgres@vm1204$ psql new_map

new_map=# \dt
               List of relations
  Schema  |      Name       | Type  |  Owner   
----------+-----------------+-------+----------
 public   | spatial_ref_sys | table | postgres
 topology | layer           | table | postgres
 topology | topology        | table | postgres

Note : I don't clean up the source files after PG install

Customize django_tinymce widget in django admin

The lazy path using an HTMLField can be customized easily:

in settings.py

TINYMCE_DEFAULT_CONFIG = {
    'plugins': "xhtmlxtras",
    'theme': "advanced",
    "theme_advanced_buttons1" : "bold,italic,underline,separator,bullist,numlist,separator,outdent,indent,separator,undo,redo",
    "theme_advanced_buttons2" : "link,unlink,separator,removeformat,separator,sub,sup,separator,abbr",
    "theme_advanced_buttons3" : "",
    'cleanup_on_startup': True,
    'custom_undo_redo_levels': 10,
}
 

all options :

• http://www.tinymce.com/wiki.php/Configuration:Advanced_theme
• http://www.tinymce.com/wiki.php/Buttons/controls

change the size of string column using sqlalchemy migrate

Why on earth have I done a table description like

CREATE TABLE address
(
  id serial NOT NULL,
  street text,
  city character varying(255),
  province character varying(10),
...

what a scrooge, 10 chars only, you can't live in saskatchewan with that. I had to migrate.

After a while I managed to get the syntax that makes it work

def upgrade(migrate_engine):
    meta.bind = migrate_engine
    address = Table('address', meta, autoload=True)
    address.c.province.alter(type=VARCHAR(length=255))

Eclipse pydev template to generate sql alchemy migrations

How often have you committed code, just to realize your missing a column in your sqlAlchemy model?
If you're using eclipse IDE, you can leverage the templates to write a migration script in no time.
Go to windows -> preferences -> PyDev -> Editor -> Templates

then create a new template with :

• name : Module: Migration SQLA
• context: New Module
• description: SQLA migration

# -*- coding: utf-8 -*-
'''
Created on ${date}

@author: ${user}
'''

from sqlalchemy import *
from migrate import *

meta = MetaData()

def upgrade(migrate_engine):
    meta.bind = migrate_engine
    some = Table('some',  meta, autoload=True)
    if not 'somename' in some.columns:
        new_col = Column('somename', Unicode(255))
        new_col.create(some)
            ${cursor}

def downgrade(migrate_engine):
    meta.bind = migrate_engine
        some = Table('some',  meta, autoload=True)
        some.c.some_col.drop()

And that's it ! you can now go into your migrations folder and do:
New -> PyDev Module, and choose your new template

default value for text function using lxml

Say we need to parse this XML

<pack xmlns="http://ns.qubic.tv/2010/item">
        <packitem>
            <duration>520</duration>
            <max_count>14</max_count>
        </packitem>
        <packitem>
            <duration></duration>
            <max_count>23</max_count>
        </packitem>
</pack>

if you want to parse it and retrieve the values in tuples

root = etree.fromstring(xml)
namespaces = {'i':"http://ns.qubic.tv/2010/item"}
packitems_duration = root.xpath('//i:pack/i:packitem/i:duration/text()', 
    namespaces=namespaces)
packitems_max_count = root.xpath('//b:pack/i:packitem/i:max_count/text()',
    namespaces=namespaces)
packitems = zip(packitems_duration, packitems_max_count)

>>> packitems
[('520','14')]

The problem is the zip result miss a value. That's because lxml returns nothing instead of None or empty string. Let's change that.

def lxml_empty_str(context, nodes):
    for node in nodes:
        node.text = node.text or ""
    return nodes

ns = etree.FunctionNamespace('http://ns.qubic.tv/lxmlfunctions')
ns['lxml_empty_str'] = lxml_empty_str

namespaces = {'i':"http://ns.qubic.tv/2010/item",
              'f': "http://ns.qubic.tv/lxmlfunctions"}
packitems_duration = root.xpath('f:lxml_empty_str('//b:pack/i:packitem/i:duration)/text()',
    namespaces={'b':billing_ns, 'f' : 'http://ns.qubic.tv/lxmlfunctions'})
packitems_max_count = root.xpath('f:lxml_empty_str('//b:pack/i:packitem/i:max_count)/text()',
    namespaces={'b':billing_ns, 'f' : 'http://ns.qubic.tv/lxmlfunctions'})
packitems = zip(packitems_duration, packitems_max_count)

>>> packitems
[('520','14'), ('','23')]

more info on extending lxml http://lxml.de/extensions.html#xpath-extension-functions