Dealing with existing data in Django

The setup can be a bit challenging if data from a previous project should be moved into the new Django project. In the Django documentation this is called integrating with legacy database. The documentation is fairly limited on this point. It took some effort to find a nice example, and some reference elsewhere on the web for this common problem.

What I did was to start with a database that already included my existing tables. With inspectdb I could define the definition of the models for these tables with Django’s best guess.

> python manage.py inspectdb > models.py
> pipenv install django_mysql

Inspectdb will write a models.py file with a Django model for each of the existing tables. The file that is created includes basic instructions for further tweaks that might be necessary. The module django_mysql defines routines to deal with the special mysql data-types enum and set.

Regular migrate and create superuser

After getting the Django description of existing data, I did a regular first migrate of the database.

> python manage.py migrate
> python manage.py createsuperuser
> python manage.py check
> python manage.py runserver

This will create the necessary tables that Django uses in the background. The command createsuperuser creates login-credentials for admin access. Login to confirm that it works.

Makemigration and migrate existing tables

In the initial migrate the models of the existing data are not included in the migration schedule yet. Makemigration will show that the newly defined tables need to be created. On regular migrate, however, this will lead to an error, since the tables already exist. It could be disaster if the existing tables were replaced by brand new ones.

The way to go is to do a fake migration to let the Django migration schedule know that the tables exist, without actually creating them.

> python manage.py makemigration
> python manage.py migrate --fake-initial
> python manage.py runserver

To show the data in admin, the models should be registered in pages/admin.py

from django.contrib import admin
from . import models

admin.site.register(models.Publication)

If the server is running, the tables will now show in admin access

Adjusting data in existing tables

If necessary you can access the models and adjust data in the existing tables. Each table is a list of objects that represent the records of the table. Yoy can get direct access to these objects in the Django-python shell. Make sure that ipython is installed, to faciltate running a small script directly in the shell.

> python manage.py shell

I started the operation with an empty field that I created in model.py with makemigrations and migrate. In regular python I wrote a routine to adapt data in one field and write the result to my new field. The routine was stored as a module in lib/my_support.py.

from pages.models import Publication
from lib import my_support as ms

pub_list = Publication.objects.all()

for i in range(0, len(publist)):
    pub_list[i].reference_cleaned = ms.clean_string(pub_list[i].reference)
    pub_list[i].save()

To run this small scipt, I copied it into the Django ipython shell and started it with ctrl-enter. From pages/models.py I loaded the table Publications. I placed the list Publications.object.all() in pub_list. The module lib/mysupport.py was imported as ms. I could now use the routine ms.clean_string() to adapt the string in the field pub_list.reference and write the result to pub_list.reference_cleaned. The result is a new filed that is filled with data that is adapted frome the initial filed. Cool.

Deploying joined tables

When deploying new data to a server location, the easy way to go is to rename the old table on the server as backup, export the new table from the local database and import in the server database. This will work fine when data are not joined. Joined data will produce an error when you try to remove the old table. As an alternative, I import an update of the table and align the two tables with basic MySql. A nice MySql-reference is MySQLtutorial.org.

The process to make the two tables similar takes three steps:
1) Make sure the two tables have the same structure
2) Make sure the two tables have the same records
3) Make sure the content of the two tables is the same

to be continued …