Task execution and automation using Invoke
a GNU Make alternative for Python projects
Very often, when working on a project, we want to run various tasks such as linting, tests, launching a development server, etc. Rather than running the actual commands to launch these tasks, it is not unusual to use some kind of tool to manage such tasks by encapsulating a bunch of commands, which can sometimes be long and complex.
Examples of such tools include grunt, gulp, npm scripts, Ruby's rake tool and GNU Make. Make is the oldest of these tools, having been around since 1976 and inspired many task execution and automation tools over the years. Many python developers still use Make today. I have seen it used on a lot of python projects, including pandas, wagtail, scikit-learn and youtube-dl. Even audreyfeldroy/cookiecutter-pypackage, a popular cookiecutter template for python packages, includes a
Makefile in the generated project!
I have also used Make a couple of times. Here's an example of a typical
Makefile I used for my LaTeX based academic assignments:
# LaTeX Makefile ## the path to your TeX file PAPER=IA_Final_Research_Proposal_Victor_Miti.tex all: pdf clean pdf: ## Compile paper arara $(PAPER) clean: ## Clean output files rm -fv *.toc *.aux *.log *.out *.blg *.bbl *.run.xml *.bcf *.fls *.synctex.gz *.fdb_latexmk
While Make is a very powerful tool, I find it rather complex, and I have not reached a point where I can comfortably construct a
Makefile to suit particular requirements. The above
Makefile is a rather simple one, but things can get much more complicated! The other problem with Make is that it was designed to work in *nix environments, so if you're using Windows, you'd have an extra job of setting up your machine to get it to work. I use Cygwin in my windows setup, so this isn't really an issue for me.
The motivation to seek a Make alternative for python projects arose when I recently started working on a FastAPI project. Being used to Django's
./manage.py runserver, I wanted a way to quickly run the development server, without having to type
uvicorn app.main:app --reload each time. Since I'm not so proficient with Make, the first idea that came to mind was to create a
runserver.sh script in the project root. However, I realized that I would probably either end up with several other bash scripts for specific tasks, or redefine
runserver.sh and write several BASH functions for a number of tasks, while providing argument variables ... it would be better to use Make instead! I decided to do a quick search on the internet for a python solution, and the following seemed to be quite prominent:
After having a quick look at the the homepages, repos and docs for the above projects, Invoke seemed to be a perfect fit for me, because of its relatively easy syntax and simple setup. So I decided to give it a shot.
As with most python packages, you can install via pip:
pip install invoke
Next, according to the docs:
The core use case for Invoke is setting up a collection of task functions and executing them. This is pretty easy – all you need is to make a file called
tasks.pyimporting the task decorator and decorating one or more functions. You will also need to add an arbitrarily-named context argument (convention is to use
context) as the first positional arg.
tasks.py file I wrote for my project (this file is located at the project root, just like a
import os from invoke import task def dev(c): """run the uvicorn development server""" os.environ["ENVIRONMENT"] = "dev" c.run("uvicorn app.main:app --reload", pty=True) def test(c): """run tests""" os.environ["ENVIRONMENT"] = "test" # setup the test database psql_command = ( 'psql -c "DROP DATABASE IF EXISTS test_postgres_database" ' '&& psql -c "DROP USER IF EXISTS test_postgres_user" ' "&& psql -c \"CREATE USER test_postgres_user PASSWORD 'testDB-password'\" " '&& psql -c "CREATE DATABASE test_postgres_database OWNER test_postgres_user" ' '&& psql -c "GRANT ALL PRIVILEGES ON DATABASE test_postgres_database to test_postgres_user"' ) c.run(psql_command, pty=True) # run pytest c.run("pytest", pty=True) def init_db(c): """use aerich to generate schema and generate app migrate location""" os.environ["ENVIRONMENT"] = "dev" c.run("aerich -c app/aerich.ini init-db", pty=True) def migrate(c): """use aerich to update models and generate migrate changes file""" os.environ["ENVIRONMENT"] = "dev" c.run("aerich -c app/aerich.ini migrate", pty=True) def upgrade(c): """use aerich to upgrade db to latest version""" os.environ["ENVIRONMENT"] = "dev" c.run("aerich -c app/aerich.ini upgrade", pty=True) def lint(c, fix=False): """flake8, black, isort and mypy""" if fix: c.run("black .", pty=True) c.run("isort --profile black .", pty=True) else: c.run("mypy app", pty=True) c.run("black . --check", pty=True) c.run("isort --check-only --profile black .", pty=True) c.run("flake8", pty=True)
Pretty straightforward, right? I was able to do this within a short time, and I know there's room for further improvement, once I dig deeper into the docs. For now, this works well for me 😄. So, if I want to run the development server, I just hit
invoke dev in my terminal. Similarly, for tests,
invoke test. Notice that the lint task has a
fix flag, whose invocation is either
invoke lint -f or
invoke lint --fix.
invoke --list or
invoke -l lists all available tasks. In this case the output is as follows:
Available tasks: dev run the uvicorn development server init-db use aerich to generate schema and generate app migrate location lint flake8, black, isort and mypy migrate use aerich to update models and generate migrate changes file test run tests upgrade use aerich to upgrade db to latest version
Notice how a task's docstring provides the task-level help! This is super cool!
Well, I'm loving Invoke so far and I think this will be my go-to task execution tool for python projects. It has an intuitive syntax, making it very easy to get up and running quickly. In addition, It is well documented and comes packed with a lot of very useful and powerful features such as shell tab completion, namespacing, task aliasing, before/after hooks, parallel execution, automatically responding to program output and more.
At the time of writing this post, I had just scratched the surface and yet I was able to achieve much! As I incorporate Invoke into my python development workflow, I will be prompted to explore it further and get to know it better. If you haven't used Invoke before, I'd suggest you give it a try and see how it fits into your workflow.