Psephology - it’s election night¶
Getting started¶
This section will get you up and running with Psephology. It is assumed that you have a working docker installed. It’s preferable to have docker-machine installed as well because it’s awesome.
Docker hub¶
Psephology can be got up and running without needing to clone the source repository since there is an image on Docker hub which is built automatically on each commit to master.
The image is configured to use the SQLite database internally and so should be considered an “ephemeral” install in that the database state is not persisted. In production, a further Dockerfile would be used which builds on this image and adds configuration for a persistent database.
Since docker will automatically pull images not available locally, we can fetch and run the Psephology server in one line:
$ docker run -it --rm --name psephology-server \
-p 5000:5000 rjw57/psephology
Although the container is now running, trying to visit the site will result in an error. This is because we’ve not yet migrated the database to the latest version. Database migration is the process of updating the schema to match the latest version. It’s good practice to have the migration be under the control of a separate command so that potentially database changing operations are explicit.
The database migration takes one command. In a separate terminal,
$ docker exec psephology-server flask db upgrade
Now you should be able to navigate to http://localhost:5000/ and see the app in
all of its glory. If you’re using docker-machine
, you’ll need to use the
appropriate IP for the virtual machine:
$ xdg-open http://$(docker-machine ip):5000 # Linux-y machines
$ open http://$(docker-machine ip):5000 # Mac OS X
Installation from source¶
Firstly, clone the Psephology application from GitHub:
$ git clone https://github.com/rjw57/psephology
$ cd psephology
One can now build the Docker container directly from the repo:
$ docker build -t rjw57/psephology .
As part of the container build, the test suite is run to ensure that the current
version is runnable inside the container environment. Once the container is
built, you can run the server using docker run
as outlined above.
Alternatively, you can opt for a local install via pip
. It is good practice
to setup a virtualenv first:
$ virtualenv -p $(which python3) venv
$ . ./venv/bin/activate
$ pip install -e .
Once installed, the test suite can be run via setup.py
:
$ python setup.py test
Code coverage can be calculated using the coverage
utility:
$ pip install coverage
$ coverage run setup.py test && coverage report
Migrate the initial database and start the server:
$ export PSEPHOLOGY_CONFIG=$PWD/config.py
$ export FLASK_APP=psephology.autoapp
$ export FLASK_DEBUG=1
$ flask db upgrade
$ flask run
When installed from source, the server is configured in “debug” mode with the Flask debug toolbar inserted into the UI. You should be able to navigate to http://localhost:5000/ and use the webapp.
Getting data in¶
We can look around the site but at the moment there isn’t much to see since there’s no data in the database. The Psephology repo comes with the results of the General Election 2017 in the correct results format. You can use the excellent httpie tool to post the results:
$ pip install httpie # if you don't have it
$ cat test-data/ge2017_results.txt | \
http POST http://$(docker-machine ip):5000/api/import
{
"diagnostics": [],
"line_count": 650
}
Note the diagnostics
field which is returned. If we add some bad results
lines then human-readable errors are returned:
$ cat test-data/noisy_results.txt | \
http POST http://$(docker-machine ip):5000/api/import
{
"diagnostics": [
{
"line": "Strangford, 507, X, 607, G",
"line_number": 1,
"message": "Party code \"X\" is unknown"
},
{
"line": "",
"line_number": 5,
"message": "Constituency name cannot be empty"
},
{
"line": "Oxford East, 11834, C, 35118, L, 4904, LD, 1785, G, 10, LD",
"line_number": 6,
"message": "Multiple results for one party"
}
],
"line_count": 7
}
We can use the API to get a table listing how many seats each party currently has:
$ http http://$(docker-machine ip):5000/api/party_totals
{
"party_totals": {
"C": {
"constituency_count": 321,
"name": "Conservative Party"
},
"G": {
"constituency_count": 8,
"name": "Green Party"
},
"L": {
"constituency_count": 263,
"name": "Labour Party"
},
"LD": {
"constituency_count": 13,
"name": "Liberal Democrats"
},
"SNP": {
"constituency_count": 35,
"name": "SNP"
}
}
}
Similarly we can retrieve the winners of each constituency via the API. Results are returned for each constituency even when there is currently no winner. (For example if a blank results line has been given.)
$ http http://$(docker-machine ip):5000/api/constituencies
{
"constituencies": [
{
"maximum_votes": 22662,
"name": "Aberavon",
"party": {
"id": "L",
"name": "Labour Party"
},
"share_percentage": 74.28459042187039,
"total_votes": 30507
},
// ....
{
"maximum_votes": null,
"name": "Belfast West",
"party": null,
"share_percentage": null,
"total_votes": null
},
// ....
{
"maximum_votes": 34594,
"name": "York Central",
"party": {
"id": "L",
"name": "Labour Party"
},
"share_percentage": 65.16350210970464,
"total_votes": 53088
},
{
"maximum_votes": 29356,
"name": "York Outer",
"party": {
"id": "C",
"name": "Conservative Party"
},
"share_percentage": 51.118811708778104,
"total_votes": 57427
}
]
}
It is also possible to update a constituency result via the API. For example, let’s allow the Liberal Democrats to win Cambridge:
$ echo Cambridge, 10, C, 10, L, 1000, LD |
http POST http://$(docker-machine ip):5000/api/import
{
"diagnostics": [],
"line_count": 1
}
Looking at the party totals, we see that Labour have lost one seat and the Liberal Democrats have gained one:
$ http http://$(docker-machine ip):5000/api/party_totals
{
"party_totals": {
"C": {
"constituency_count": 321,
"name": "Conservative Party"
},
"G": {
"constituency_count": 8,
"name": "Green Party"
},
"L": {
"constituency_count": 262,
"name": "Labour Party"
},
"LD": {
"constituency_count": 14,
"name": "Liberal Democrats"
},
"SNP": {
"constituency_count": 35,
"name": "SNP"
}
}
}
Building the documentation¶
The documentation is built with the sphinx
tool and has additional
requirements. You can install the requirements and build the documentation via:
$ pip install -r doc/requirements.txt
$ make -C doc singlehtml
$ xdg-open doc/_build/singlehtml/index.html # Linux-y
$ open doc/_build/singlehtml/index.html # OS X
Web UI¶
The web UI is available at http://localhost:5000/ (or at the appropriate IP if
using docker-machine
). The following pages are available:
- A summary giving total number of seats for each party
- A list of winners for each constituency
- An event log showing any errors/warnings from importing results files
- A page which lets the user upload a new results file
- A page which provides the current results as a plain text file in the result line format
Command line¶
The Psephology application has a command-line interface which can be run via the
flask
tool. Perhaps the most useful command is flask run
which will
launch a server hosting the application.
Database migration¶
Psephology uses Flask-Migrate and alembic to manage the database migrations. On
first run or when upgrading the software, remember to run flask db upgrade
to migrate the database to the newest version.
Importing results¶
Results may be imported from the command line via flask psephology
importresults
. See flask psephology importresults --help
for more
information.
Programmers’ reference¶
Application object creation¶
The app
module provides support for creating Flask Application
objects.
-
psephology.app.
create_app
(config_filename=None, config_object=None)¶ Create a new application object. The database and CLI are automatically wired up. If
config_filename
orconfig_object
are notNone
they are passed toapp.config.from_pyfile()
andapp.config.from_object()
respectively.Returns the newly created application object.
The autoapp
module may be imported to automatically create an
application from the configuration specified in the PSEPHOLOGY_CONFIG
environment variable. This is useful, for example, when using the gunicorn
web server where one can launch the application via:
$ gunicorn psephology.autoapp:app
Importing/exporting data¶
The io
module provides functions which can be used to parse external
data formats used by Psephology.
-
psephology.io.
parse_result_line
(line)¶ Take a line consisting of a constituency name and vote count, party id pairs all separated by commas and return the constituency name and a list of results as a pair. The results list consists of vote-count party name pairs.
To handle constituencies whose names include a comma, the parse considers count, party pairs from the right and stops when it reaches a vote count which is not an integer.
Data model¶
The model
module defines the basic data model used by Psephology
along with some utility functions which can be used to mutate it. The
query
module contains some potted queries for this data model which
provide useful summaries.
None of the functions in model
will run session.commit()
. If you
mutate the database inside a UI/API implementation, you’ll need to remember to
commit the result. This is to guard against partial updates to the DB is a
UI/API method fails.
-
class
psephology.model.
Party
(**kwargs)¶ A political party. Each party is primarily keyed by its abbreviation. In addition, the name of a political party should be unique.
-
id
¶ String primary key. This is the party “code” such as “C” or “LD”.
-
name
¶ Human-readable “long” name for the party.
-
-
class
psephology.model.
Constituency
(**kwargs)¶ A constituency. Essentially this is a mapping between a numeric id and a human-friendly name.
-
id
¶ Integer primary key.
-
name
¶ Human-readable name. Must be unique.
-
-
class
psephology.model.
Voting
(**kwargs)¶ A record of a number of votes cast for a particular party within a constituency.
-
id
¶ Integer primary key.
-
count
¶ Number of votes cast.
-
constituency_id
¶ Integer primary key id of associated constituency.
-
constituency
¶ Constituency
instance for associated constituency.
-
party_id
¶ String primary key id of associated party.
-
-
class
psephology.model.
LogEntry
(**kwargs)¶ A record of some log-worthy text.
-
id
¶ Integer primary key.
-
created_at
¶ Date and time at which this entry was created in UTC. When creating an instance, this defaults to the current date and time.
-
message
¶ Textual content of log.
-
-
psephology.model.
log
(message)¶ Convenience function to log a message to the database.
-
psephology.model.
add_constituency_result_line
(line, valid_codes=None, session=None)¶ Add in a result from a constituency. Any previous result is removed. If there is an error, ValueError is raised with an informative message.
Session is the database session to use. If None, the global db.session is used.
If valid_codes is non-None, it is a set containing the party codes which are allowed in this database. If None, this set is queried from the database.
The session is not commit()-ed.
-
class
psephology.model.
Diagnostic
(line, message, line_number)¶ A diagnostic from parsing a file. Records the original line, a human-readable message and a 1-based line number.
-
psephology.model.
import_results
(results_file, valid_codes=None, session=None)¶ Take a iterable which yields result lines and add them to the database. If session is None, the global db.session is used.
If valid_codes is non-None, it is a set containing the party codes which are allowed in this database. If None, this set is queried from the database.
Note
This can take a relatively long time when adding several hundred results. Should this become a bottleneck, there are some optimisation opportunities.
The query
module provides some ready-to-use queries which can be run
against the database.
-
psephology.query.
constituency_winners
()¶ A query which returns the Constituency, Voting, winning vote count and total vote count for each constituency. If there was no winner in the constituency, then the Voting, winning vote count and total vote count is None.
The maximum vote count for a constituency is labelled ‘max_vote_count’ and the total vote count is labelled ‘total_vote_count’.
If you intend to get related objects from the Voting, make sure to add an appropriate joinedload() to the options.
E.g.:
q = constituency_winners().options( joinedload(Voting.party) )
-
psephology.query.
party_totals
()¶ A query which returns a Party and a constituency count, labelled ‘constituency_count’ which gives the number of constituencies that party has won.