How to find a good freelance developer

A bunch of people have asked me recently how to find good contract developers. I’ve done this quite a few times now and feel like I now have a pretty good system for selecting quality freelancers at good fixed prices.

I can tell that this system works because on my last two contract projects I received an average of 100 bids per posting (spread across 3 different sites) and both hires ended up blowing me away with the quality, speed, and price of their work. I have viewed many friends’ postings that have received 10 or fewer bids, many of which were low quality. I’ll take you through what I do to give myself better developer options.

Step 1

Write a great specification of the work you want done. There are a lot of reasons why this is important.

First, the best freelance developers are more likely to place bids if they know the scope of the work. When they don’t know the scope of the work it becomes dangerous for them to place a fixed price bid because they might end up doing far more work than they had hoped. These developers care a lot about their aggregate ratings on these freelance sites and will make sure they do everything in their power to make their customer happy and get a top rating. If you lay out a detailed specification, they know exactly what it will take and are no longer scared that you have a list of features up your sleeve that you want them to implement as well.

Basically a great specification will tell the potential hire that you know what you’re doing and that you will be pleasant to work with.

Second, you will need to define all of these features at some point anyway so you might as well do it right up front in order to gain the benefit of finding the best possible developer. It’s a good exercise to fully define your product as you will find tricky parts you hadn’t thought of while simply thinking about your product at a high level. The earlier you think about and address these tricky bits, the less costly it will be to implement well.

Here’s an example of a spec I wrote recently that received 110 bids, at least 8 of which were very high quality: https://docs.google.com/document/d/15c6n6HAUoa5SDP7TlGU_sljpdIA_8doe3h2Kbi9VcoM/edit

Step 2

Post your spec to elance.com without setting a price. They allow you to not specify your budget or expected price. By posting it without a price you can wait for the bids to roll in and you’ll start to get a good idea of how much this job should cost.

Step 3

Take the expected cost determined in step two and post the same spec to odesk and freelancer.com. There are plenty of other freelance sites out there so you are free to post to those as well. I have had great luck with those three sites (elance, odesk, and freelancer) and can confidently say there are some very high quality developers on each of these sites who will come out of the woodwork if your spec is good.

Step 4

Wait. It’s tempting to hire the first competent person who submits a good bid, but you will get plenty more if you wait. Be patient and wait for the bids to start slowing down. 2-3 days should be enough.

Step 5

Narrow down the bids to find the right developer. This can seem overwhelming at first but if you do it systematically you can do it in about 2-4 hours. Here’s what I do:
  • Narrow down the field based on their bid price, their quantitative ratings, and the amount of work they’ve done in the past. After keeping only people with 4.9+ ratings, a significant amount of work, and a very good bid price the field usually narrows to about 10.
  • Briefly glance at each of the top 10 applicants’ portfolios and take a look at their communication skills in their messages with you or on their profile page. Eliminate all but the very best. Hopefully you’ll be down to your last 2-4 people now.
  • Choosing from these last few is the toughest step and for this I tend to choose the person I think would make the best long term hire. English speaking ability and the general vibe I get from the person play a big role here. Of course this is just a contract project, but while working with this person you will build up a relationship and if they produce high quality work the chances are you will want to hire them for work in the future as well.

Final Note

If your project is big you may want to break it up into small pieces. For instance you can write the spec for just a small part of your application so if you end up hiring someone who produces low quality work, you won't lose much because you haven't invested much. I view the first job I award someone as an interview to some extent. If you like that person you can always hire him/her for the rest of your project.

Good luck!

Why are you doing this?

One of the mentors at The Iron Yard this summer asked the question "Why are you doing this?". I love this question because without a really good reason for doing this, building a company becomes infinitely harder. Recently I had one of my most grueling days since I started Leaguevine when I spent the entire day on the phone making sales to people who mostly didn't give a crap. I don't like sales. But I could force myself to work hard and get stuff done because I have a very clear end goal.

I want to have a huge, positive impact on the world through international development and to do so requires a large amount of money. This is at the top of my mind every day and has been sitting there for about two years.

When I was devoting a vast majority of my energy to Engineers Without Borders from mid 2008 through 2010, my view on the world changed completely. Before EWB, I would have been content with a comfortable life where I was able to travel a lot and have some fun. I could work for a great employer, produce at a high level, get paid a lot, and do the things I wanted on nights and weekends. 

EWB killed this former dream life for me. I am certain that I would no longer be content with such a life. 

After my first trip to Orongo, Kenya and seeing all of the challenges people have to go through every single day, I returned home grateful for running water. For a full year after that first trip I would think about how lucky I am to have running water every time I turned on a faucet. 

On my second trip I was given a grand tour of Kibera by one of its residents. Kibera is one of the largest slums in the world and I was walking through it shortly after it had rained. Walking through this town was excruciatingly difficult because all roads and pathways were dirt and because many of these pathways were steep and it had rained, it was really tough walking around. I remember Cartoon (my friend in Kibera) reaching down and extending me a hand to help me climb various pathways. Not surprisingly, I thought about Kibera at least once a day for the entire calendar year of 2009.

By the fourth time I traveled to Orongo, Kenya I realized how much I loved doing this International Development work. Every day I was using my knowledge, connections, and hustle to make the most change I could in this charming little community. Despite loving this work, I realized how many parts of this system were broken. I don't mean to say EWB or other NGOs are bad, but the entire state of International Development could use a heap of improvement. Because of this, I would not be content being a cog in this big International Development machine where I would likely have very little meaningful impact over a lifetime of hard work.

At this point, because I was no longer content with a simple software developer's life and I would not be content working for an NGO in this International Development space, the path to reaching my dreams started to become somewhat clear. I would have to do something that gave me a chance to make a huge amount of money which I could then use to instill my own vision into the International Development scene. 

I like to think of this as the Bill Gates model of doing good in the world. Step 1 is to make a lot of money, and step 2 is to use the money to do something incredible. Gates is doing far more than just donating his money to charity. He is aggressively attacking the world's problems with the same gusto that he used to build Microsoft by defining his own well-rounded initiatives and finding the right partners to carry out his vision. He's crushing it. He is not conforming to standard NGO models, and is willing to disrupt this space and do whatever it takes to improve the world. And for that he's my idol.

While it's unrealistic to expect to make an amount of money and impact on the same order of magnitude that Gates did, this general model for living my life is well within the realm of possibility. This is what gets me fired up every morning. And more importantly, it's what keeps me going at full speed when things look bleak.

Fixing tech startup incorporation documents

A year ago I incorporated Leaguevine myself without using any lawyers. I wrote about it here and heavily relied upon the Orrick company document templates. I realized back then that there was a good chance I would screw some stuff up and decided it would be better for Leaguevine to first get some funding or free access to lawyers and at that point revise anything I messed up.

Turns out this approach worked great.

I am currently attending The Iron Yard accelerator program in Greenville, SC and this program gave us not only funding and connections, but also free legal services from a quality law firm called Wyche. Our lawyer there took care of all the amendments where necessary for free and now we have a great set of documents. He told us that the templates that we used were excellent and far better than a lot of the templates he sees self-incorporators use. So I'm glad we used the Orrick templates and saved him a lot of work.

I'm really happy I decided not to pay for legal services at the very start of this company.

Python syntax highlighting in Django templates using Pygments

Highlighting code to be placed on your site is a pretty common need that has been addressed nicely by a number of open source projects. Pygments is great, but on it's own would take you a while to get set up. This guide is meant to get you set up with using pygments in your django templates in under 5 minutes.

Choose your syntax highlighting style

First, decide which pygments style you want to use by trying them out on the pygments site.

Install django-pygments

There's a nice project on github that gives you a couple template tags for using pygments.

Create a pygments css file

Using whichever  style you liked from the pygments site, you can use the pygmentize command in your terminal to generate a CSS file for marking up your code.

pygmentize -S vs -f html > vs.css

Mark up your django templates

You're now ready to make the changes to your templates. Just import your css file, load the template tags, wrap your pre-formatted code with the pygment template tag, and then label the code as whatever language you use.

<link href="/media/css/pygments/vs.css" media="screen" rel="stylesheet" type="text/css" />

{% load pygmentify %}
{% pygment %}
<pre lang="python">
import simplejson
import urllib2
</pre>
{% endpygment %}

Almost 4 months in

I've been working full time on Leaguevine for almost 4 months now, and things are finally beginning to click for me. It's a completely different lifestyle, and I'm finally accepting the fact that it takes a while to adapt to it. Things like self-motivation, prioritization, setting deadlines, and staying organized are all things that I never had to deal with before since school basically took care of all that for me. These tasks are becoming easier every day, and I wouldn't trade this experience for anything at this point!

A couple nights ago I realized just how much I love what I'm doing. I was hanging out with friends who were back in town for Thanksgiving until 2:30am, and despite having a blast catching up with those guys, all I could think about on my drive home was how pumped I was to finish implementing our oauth2 server for our API. Indeed, I went right to work when I got home (don't worry - I hadn't been drinking with my friends) and worked until I was just too exhausted. I went to bed really happy.

We have yet to release our core product to the world, and won't do so for another few months, but I imagine once all our cool stuff goes live and we have more users, this experience will only get better. 

Reducing CrashPlan's Disk Space Usage on OSX

I have CrashPlan backing up almost every file on my Hackintosh, and when CrashPlan runs, it caches a ton of data so it can run faster. However, they don't let you configure where that data is stored. In my case, I'm backing up around 1TB of data, and thus the cache is a whopping 8GB. 8GB might not seem like that much these days, but when it's sitting on your small SSD, it's quite annoying. Fixing this problem was pretty easy, and since plenty of people were complaining about this on their forums in different threads, I thought I'd post my simple solution here.

First, you need to have a second drive connected to your system that you'd rather have hold the cache. If you have this, then this solution should work for you:

1. Close CrashPlan, as you normally would.
2. Stop CrashPlan's background service by typing: 

sudo launchctl unload /Library/LaunchDaemons/com.crashplan.engine.plist 
 
3. Remove the Cache:

rm -r /Library/Caches/CrashPlan

4. Create a simlink: 

ln -s /Volumes/OtherDrive/path/to/new/cache/ /Library/Caches/CrashPlan

5. Start CrashPlan's background service: 

sudo launchctl load /Library/LaunchDaemons/com.crashplan.engine.plist

6. Restart CrashPlan

This should free up a whole bunch of space on your startup disk. Enjoy!

Django-Celery on Webfaction using RabbitMQ

This tutorial is meant to get you up and running from scratch with django-celery on Webfaction. Each of the steps is a bit of a hassle since you typically need to find different install steps for each individual part, so I just lumped up the whole experience in this guide.

Install Erlang

Erlang is needed for installing RabbitMQ which is the preferred message broker for Celery. Webfaction doesn't come with this installed, so you'll need to do it manually:
  • Go to the webfaction control panel and create a new app -> Custom App, Listening on Port
  • Download the latest version of Erlang. You can just use the command: wget http://www.erlang.org/download/otp_src_R14B03.tar.gz
  • Unzip it: gunzip -c otp_src_R14B03.tar.gz | tar xf -
  • cd into the directory
  • Configure the build: ./configure --prefix=/home/your_webfaction_username/
  • Make it: make
  • Install it: make install
  • Run it on the port given to you when you created the new Erlang app: epmd -port 12345 -daemon
Install RabbitMQ
Next, you need to change the file ~/lib/rabbitmq/rabbitmq-server. I found some information about this on the webfaction community forums. Open your text editor and change three lines to:

CONFIG_FILE=~/src/rabbitmq_server-2.6.1/sbin/
LOG_BASE=~/logs/user/rabbitmq
MNESIA_BASE=~/src/rabbitmq_server-2.6.1/sbin/

Added these lines to the rabbitmq-env file and use the ports you reserved for epmd and rabbitmq in your earlier steps:

export ERL_EPMD_PORT=12708
export RABBITMQ_NODE_PORT=35478
export ERL_INETRC=$HOME/.erl_inetrc

Added the file $HOME/hosts which looks like:

127.0.0.1 localhost.localdomain localhost
::1      localhost6.localdomain6 localhost6
127.0.0.1 web160 web160.webfaction.com

Added the file $HOME/.erl_inetrc which looks like:
 
{hosts_file, "/home/<your_user_name>/hosts"}.
{lookup, [file,native]}.

Run Rabbitmq and check that it is working:

./rabbitmq-server -detached
./rabbitmqctl status

Finally, add a new user and vhost, and configure it so only your app will have access to it.

./rabbitmqctl add_user <username> <password>
./rabbitmqctl set_user_tags <username> administrator
./rabbitmqctl add_vhost <vhostpath>
./rabbitmqctl set_permissions -p <vhostpath> <username> ".*" ".*" ".*"
./rabbitmqctl clear_permissions -p <vhostpath> guest

Install Celery and Django-Celery

pip install django-celery

In your settings file, you will then need to add the lines:

BROKER_HOST = "localhost"
BROKER_PORT = 36784
BROKER_USER = "username"
BROKER_PASSWORD = "password"
BROKER_VHOST = "vhostpath"
CELERYD_CONCURRENCY = 1
CELERYD_NODES="w1"
CELERY_RESULT_BACKEND="amqp"

The reason we set the concurrency so low is because Celery takes up a good amount of memory, and you are likely limited with your memory consumption on webfaction. The minimum amount of memory Celery can take will be however much it needs to run the main process (consuming messages, sending tasks to workers, etc), and a worker tasks that actually does stuff. Each of these will take up about 20-30MB of memory depending on the size of your Django app.

Add 'djcelery' to your installed apps.

Follow any other steps listed in their installation guide that are relevant to your app.  If you are using mod_wsgi, add the following to your .wsgi module:

import os
os.environ["CELERY_LOADER"] = "django"

Install a tool to create a Daemon

Celery does not daemonize itself, and thus you need to do this yourself. Creating a daemon is not exactly the same as simply running it in the background, so you should install a tool that can help you do this. Celery recommends a couple options. One of the easiest ways is to use a simple tool called django-supervisor. To install this, just type: 

pip install django-supervisor

Add the file supervisord.conf in the same directory as manage.py, and add the content:

[program:celeryd]
command={{ PYTHON }} {{ PROJECT_DIR }}/manage.py celeryd -l info

[program:autoreload]
exclude=true

[program:runserver]
exclude=true

[program:celerybeat]
exclude=true

Every time you restart your webserver, you can restart celery by issuing the following commands:

python manage.py supervisor --daemonize
python manage.py supervisor stop all
python manage.py supervisor start all

However, there is a downside with using django-supervisor in that it will run in the background and take up another 20-30MB of memory. A more memory efficient way would be to install a tool called daemonize. This page has very easy installation instructions. Once you install it, just add an alias to it in your .bashrc or .profile and then run:

daemonize /<full_path_to_django_directory>/manage.py celeryd

Everything should then be up and running. Good luck!

Self-Incorporating a Tech Startup

This is in response to my friend Ben's recent post, What about money for a lawyer?

I spent a bit of time last month self-incorporating my tech startup and since this topic is very fresh on my mind, I thought I'd reflect on it for a moment.

The first thing I'd like to reflect on is the actual decision about whether or not to do this myself or hire a lawyer. The decision here comes down to deciding whether or not the additional $1,200-$1,500 in fees is worth the peace of mind that your company is starting off on the right foot in the legal department. Since I am confident that I know what I'm doing and have free resources to monitor I'm not doing anything dumb, the decision to by-pass hiring an attorney was an easy one. And I believe any competent entrepreneur can do the same, if they are willing to give up some time and energy to educate themselves. The trick is to get to the point where you can be confident that you are doing it right without missing any steps. So that's what I'll focus on here.

To figure out what all goes into forming your company, there are more than enough free, reliable, online resources to help educate you. Just use google to find legal articles on this stuff, and then study wikipedia to make sure you understand all of the legal terminology.  A few good blogs for learning about this stuff are Startuplawyer, TechnologyStartupLaw, Startup Company Lawyer, Brad Feld's Blog, and of course the MBA Monday's series on AVC. From reading these sites as well as learning from Quora and Stack Exchange's Onstartups, it was clear to me exactly what details I wanted to bake into incorporating my company. I won't talk about legal specifics here, and I'll just talk about how I learned enough to incorporate everything myself.

So knowing all of the details about type of entity, state of incorporation, par value, vesting agreements, IP rights, etc, is a good start, but the more intimidating part is actually going about drafting and filing the appropriate legal documents to put everything in place. The best way to do this, if possible, is to use free resources to legal assistance. You'd be shocked at how many people out there really want to help you. Two good options for this are:
  1. Free legal clinics. Tons of Universities with Law Schools have these now, and they are aimed at giving law students real world practice with startup law while being supervised by professionals. In Chicago, the top three are the program at Northwestern University, the Institute for Justice Clinic, and the Loyola Law Clinic. A lot of these will have long wait times, so if it's not urgent these could be a great free source of help.
  2. Small Business Development Centers (SBDCs). These will often give you general business advice for free, and have connections to pro-bono legal help or Certified Public Accountants (CPAs) who are plenty knowledgeable about simple legal issues. I went into the SBDC at UIC and was happy to get a lot of help from them. In fact, I still email the CPA now and then if I have a quick question and he always sends me back a clear response in an hour or two.
Once these two options for getting help with filing your documents have been explored, it is worth also checking with lawyers to make sure you are doing this right. Lawyers will all give you a free consultation half an hour or so to learn more about you and attempt to woo you into signing with their practice. I talked on the phone with about 8 lawyers for 30 minutes each, about setting up my corporate entity and drafting other legal documents. At first, I was embarrassed to say I was thinking about incorporating on my own without their help because I thought they would call me naive or say I shouldn't do it. But after the 2nd interview, I realized that these lawyers were actually enthusiastic about the idea of me saving money for my company upfront and they were more than helpful to me in hopes that I would sign on with them when I had "real" legal needs that needed to be addressed such as raising a round of capital. These lawyers would all give me lists of legal documents I needed to file, tips on how to file them and what I should incorporate on them, and told me about any pitfalls or things to watch out for (such as making sure I file my 83(b) within 30 days of incorporation). They assured me that internal documents such as company by-laws can be amended later when you need to, so getting it perfect the first time will not make or break your company. Further, one lawyer even offered to inspect my documents for free after I drafted them to ensure I was doing things right.

So I guess what I learned is that incorporating a business is not hard, and you can learn everything you need by just using Google, calling people, and asking the right questions. Since in my situation I have a lot of time but not a lot of money, this made sense. While all of this information was a bit overwhelming at first, I now feel infinitely more confident with my business since I know the ins and outs of my company's legal structure.

One month down

It's been one month since I started working on Leaguevine full time, and it's been a very interesting experience. After being in school for 20 straight years, being on my own has been quite a shock. I now set my own schedule and am accountable only to myself. As grand as all the blogs out there make that seem, it's kind of weird actually being in the middle of it.

While I was in school, I would always have my side projects and activities that I actually cared about far more than school. While I spent most of my time on these outside activities, school has always had a constant presence. It has been an anchor and something that brings consistency to all the years of my life, even if my outside activities change. I'd become conditioned to working quickly and efficiently on my schoolwork so I could make time for my passion projects. For the last several months, I kept saying to myself how much I'd love to work solely on Leaguevine instead of having to also do relatively meaningless class work and research. But then August 1st rolled around and I didn't get into the sprint I was hoping for.

I found that during my first few weeks I often became overwhelmed into complacency. Instead of instantly working 80 hour weeks like I had planned, I was putting in 50-60 tops and couldn't really bring myself to do anymore even though I had nothing else to work on. I would take long breaks to watch youtube or play Starcraft II. And it felt like a huge waste of my time. Mostly, I felt bad that I wasn't giving the business my undivided attention.

I was being overwhelmed by a number of different things. First, there were no "real" deadlines in sight anywhere. Every deadline was just lying on a piece of paper or my google calendar and I kew I wouldn't be punished if I missed it by a few hours, days, or even weeks. Next, I had just come back from a fantastic trip to China, and had been living in "consumer" mode. I had to go back to the mindset of "what can I build today" rather than "what can I buy today". I think the thing that had the biggest stress on me was incorporating my business. I talked to about 8 lawyers, a number of non-profit clinics, a CPA, my dad, and an SBDC about getting the company set up right, and all this talking on top of reading blogs about tech startups made me feel like I wasn't getting anywhere. I was spending so much time working on the business plan and worrying about legal stuff, that I was barely writing any code. And this lack of progress on the web app just made me feel like I was falling farther behind and wasting my time.

I'm not sure exactly what happened, but sometime in the last week or so a light went on and I've been fully focused on Leaguevine. I think it's because I finally got back to coding. I've become okay with the notion that this is going to be a long process that will take time, and I'm finally focusing on what I can accomplish each day and just doing that. I've been working 12+ hours a day every day this past week and I'm loving a vast majority of it. Yes, there are times when I have to do some tedious stuff, but the percentage of time in a day where I am truly enjoying what I do is higher than it was while I was in school.

I'm not sure exactly how much this has contributed to this productivity improvement, but since I had been the most lazy in the mornings, I came up with a little trick to make myself more productive right after waking up. Before going to sleep, I'd open up one of the source code files and begin adding a single line of code to it. I don't finish the line, and it doesn't even have to do anything useful, but I then turn off my monitor without even saving the file. As is typical with VI guys, I can't stand files being opened and unsaved, so the first thing I do in the morning is delete that line of code or finish it. Just this one little 3 second action gets me in the mood to develop something cool, and immediately I'm working at full speed the moment I sit down at my computer. This tip goes back to the philosophy of beating procrastination by

Read the rest of this post »

Migrating a Django Postgres DB from Concrete Inheritance to Abstract Inheritance

Django comes with several ways of implementing model inheritance, and specifying which one you would like to use takes only a line or two of code. Setting up your database for the very first time is extremely easy, but migrating between types after you have existing data in infinitely harder. I have a live site whose needs have changed since I built the database and the concrete inheritance I set up is no longer needed. In my case, the site should be using Abstract Inheritance

There are a number of gotchas in this process, and I will outline exactly how I navigated my way through this process. First, I should note that I use django-south in my project and the migrations rely heavily on using this. I asked for advice on Stackoverflow  and I'll use a similar example here. This article will walk you through the process of migrating from the before to after schema while keeping all your data intact.

Before:

app1/models.py:

    class Model1(base_app.models.BaseModel):
        field1 = models.CharField(max_length=1000)
        field2 = models.CharField(max_length=1000)
    
app2/models.py:

    class Model2(base_app.models.BaseModel):
        field1 = models.CharField(max_length=1000)
        field2 = models.CharField(max_length=1000)
    
base_app/models.py:

    class BaseModel(models.Model):
        user1 = models.ForeignKey(User, related_name="user1")
        user2 = models.ForeignKey(User, related_name="user2")
        another_field = models.CharField(max_length=1000)

        objects = CustomManager()

After:

app1/models.py:

    class Model1(base_app.models.BaseModel):
        field1 = models.CharField(max_length=1000)
        field2 = models.CharField(max_length=1000)

        objects = CustomManager()
    
app2/models.py:

    class Model2(base_app.models.BaseModel):
        field1 = models.CharField(max_length=1000)
        field2 = models.CharField(max_length=1000)

        objects = CustomManager()

    
base_app/models.py:

    class BaseModel(models.Model):
        user1 = models.ForeignKey(User, related_name="%(class)s_user1")
        user2 = models.ForeignKey(User, related_name="%(class)s_user2")
        another_field = models.CharField(max_length=1000)
    
        class Meta:
            abstract = True

These are the models I will use to walk you through this migration process. Essentially what we are doing here is removing the one-to-one relationships between Model1/BaseModel and Model2/BaseModel and instead placing each of the fields of BaseModel into Model1 and Model2 so they actually reside in the tables for Model1 and Model2. The tricky part is migrating all your data while we work through this.

Make a Copy of BaseModel

When you add the "abstract = True" property to the BaseModel (don't do this yet), south will delete the entire table BaseModel. If you don't make a copy of the data in BaseModel, after you add the "abstract = True" property you will have no way of copying the fields in the old BaseModel to the respective new models. Making a copy isn't bad:
  1. Add BaseModelCopy within base_app/models.py that has an identical schema to BaseModel.
  2. Run a schema migration (python manage.py schemamigration base_app --auto; python manage.py migrate base_app).
  3. Run a data migration to copy the existing objects in BaseModel to BaseModelCopy (python manage.py datamigration base_app copy_contents; (then edit the contents of the data migration); python manage.py migrate base_app).
Add a Field on the Child Classes to Store the ID

The way django deals with concrete inheritance is a little interesting when you look closely at it. The Child classes Model1 and Model2 do not have 'id' columns in the database, and their primary_key is actually a column named "basemodel_ptr_id". Thus, the unique id is stored in the corresponding BaseModel object. When you eventually add the "abstract = True" property, south will naturally delete this "basemodel_ptr_id" column and you will lose all references to the original BaseModel object. This is clearly very bad, and we would have no way of recovering those relationships if we did that. Thus, we need to add a field to Model1 and Model2 that stores the id of the corresponding BaseModel object so we can reference it in the data migrations after we move to abstract base classes. Doing this is also straightforward:
  1. Add a field "tmp_id = models.IntegerField()" on both Model1 and Model2.
  2. Run a schema migration on app1 and app2
  3. Run a data migration on app1 and app2 copying the id of the corresponding base model into the "tmp_id" field.
Prepare the BaseModel to be Abstract

There are two things you will need to change in your BaseModel class before you can add the "abstract = True" property. First, you will need to fix the related_name for every ForeignKey field in BaseModel. If you don't do this, there will be multiple tables with ForeignKeys on the User object with the same related_name and this conflict will cause an error. Adding the name of the class to the related name as shown in the "after" state of the database will solve this issue.

The second issue is that you will have to remove any managers on BaseModel. These managers will throw an error and instead need to be bound to the child classes.

Mark the BaseModel as Abstract and Create a New Primary Key

There is yet another gotcha with this step. When we add the property "abstract = True" to BaseModel, after django deletes this model, it will automatically add an "id" field to Model1 and Model2 and it will make this field a primary_key. This is fine if we have no data, but since we do have data, we have no way of specifying what to set these initial "id"s to, and since it is a primary_key and there can't be any overlaps, we simply can't do this. Conveniently, we have this "tmp_id" field full of unique values, and we can use this as our primary_key. We can specify this when we make our migration:
  1. Add the property "abstract = True" on BaseModel.
  2. Add a field "id = models.IntegerField(primary_key=False)" on BaseModel
  3. Change the "tmp_id = models.IntegerField()" on Model1 and Model2 to be "tmp_id = models.IntegerField(primary_key=True).
  4. Run schema migrations on app1, app2, and base_app.
  5. Run a datamigration that copies the relevant data from BaseModelCopy to every corresponding object in Model1 and Model2. To do this, just make use of the "tmp_id" field we created and the new BaseModelCopy class. Make sure this datamigration also copies the "id" field.
Remove all our Temporary Models and Fields

Our database should be near-perfect now and you should be able to run the development server to view your site. Inserts will not work yet because we have not specified a way to auto-increment the primary key of objects when we insert them, but we'll get to that later. Right now, if we see all our data has migrated successfully, we are ready to remove all this temporary stuff we created.
  1. Remove the "tmp_id" columns from Model1 and Model2. 
  2. Change the "id" field on BaseModel to "id = models.IntegerField(primary_key=True)".
  3. Remove the model BaseModelCopy.
  4. Run schema migrations on app1, app2, and base_app.
Build the AutoField Functionality on Your Primary Key

The final gotcha of this process is that Postgres handles the django field models.AutoField() a bit strangely. You can inspect this using pgadmin or whatever gui you have to look at how it adds a "serial" property to the field. Because of this peculiarity, south will not allow you to migrate from an IntegerField() to an AutoField() out of the box. There is a workaround for this migration, but if that seems to messy to you, you can simply handle the auto-increment manually. Handling it manually only takes a couple of lines, but make sure you implement some sort of database locking to ensure you don't run into race conditions.

If you are not using Postgres, you may be able to completely skip this step and instead just change the id field on BaseModel to "id = models.AutoField(primary_key=True)" and run a migration, but I can't confirm this.

Push Changes to your Production Server

I'm assuming you made all of these migrations on local servers and not a production server, so the final step is to make the jump to deploying the changes. We now have our new schema with all the data filled in, and it should be working nicely with our code base on our local server. However, our set of migrations poses some problems for when we migrate. We can see that each set of migrations we did in the steps above are reliant on the full set of migrations having been completed in the previous step. This means the migrations for one app are reliant on the migrations for another app so going through a full set of migrations for a single app would not work. 

To avoid this issue, we could have pushed each individual set of migrations along with the code to the production server right as we generated them. Alternately, we can simply clear out the migration history for each of our apps, dump the local datastore, and then rebuild the production database using these migrations. This is the method I used because I happened to be doing this at 4am when no one users were generating new data on the stie.

So, these are the steps for how to do this:
  1. On your local server, remove all of the migrations from from migrations/ folder for app1, app2, and base_app. The history is no longer important to us because we did not specify a way to do backwards migrations anyway.
  2. On your local server, create initial migrations using south for app1, app2, and base_app, clearing any ghost migrations (python manage.py schemamigration app1 --initial --delete-ghost-migrations).
  3. On your local server, run the migrations. South will tell you that nothing needs to be changed. This is good.
  4. Save all your changes to git or whatever you use.
  5. Dump the local database (pg_dump --no-owner --no-acl -U username postgres_db_dev > postgres_db_dev_dump.psql).
  6. Clone your working repository onto your production machine.
  7. Copy your sql dump to your production machine.
  8. Delete the production db (dropdb -U username postgres_db).
  9. Re-create the db (createdb -U username postgres_db).
  10. Load the new data (psql -U username postgres_db < postgres_db_dev_dump.psql)
Now your production server should look identical to your local_server that you had a working version on. Now that they are synced, you can go back to using whatever regular deployment scripts you use and everything should be just fine.

Conclusion

The methods used in this tutorial are useful if you have a live server with data you don't want to lose, but also have a span of time where you can be confident that users will not be adding new data. If users had entered new data into the production site during this process, their data would be lost during the final deployment to the production server. My site has negligible traffic from 1am-7am when I did this, and the database is small (<50,000 objects of type BaseModel) so I could do this quickly. To overcome this on a medium-high traffic site that uses user generated data, you should really schedule some "read-only" downtime and alert your users of this to ensure they don't enter data that gets overwritten.