Table partitioning

Overview

The measurements table is too large is now being partitioned by the day. Each day a manager called create_table_partition is called python manage.py create_table_partition

create_table_partitions ensures there are fifteen daily partitions beyond the current day's table. If there is an error, an email will be sent out notifying the admin

Currently the process is part of a script that considers the virtualenv of the production environment, but since this script exposes path names it is gitignored.

Once configured table partitioning is transparent to the end user. At the prompt you will still use table name measurement_measurement and not directly select from a partition. The Django ORM is not aware of the partitioning either.

Migration process

In order to convert the existing table to a partitioned table the following steps were taken. See scripts in scripts dir

Copy schema from existing table:

1. Migrate db schema and all data except measurements to RDS instance
2. Save existing measurements schema to file named measurement.sql
3. Drop measurement table
4. Locally run scripts/migrate_measurements.sh which uses docker postges 12.4. See script for details