Building a Template for a Network Automation Project
Using templates for device configurations is a common practice and it has obvious benefits, such as speed and consistency. Working on many small Python automation projects made me think of employing the same approach. Previously I had to copy and adjust a lot of code-related things such as directory structure, poetry settings, CI/CD pipelines, etc. Templating all of this allowed me to reduce the initial scaffolding overhead to a minimum and jump straight into writing code. In this article, I want to share my experience in building such a template.
But first, let's focus more on the whys of project templating. Here is what I make of it:
- It brings consistency to the projects, especially when working in a team (and there are always at least you and your future self).
- It enforces the use of best practices, such as linting and formatting.
- It saves a lot of time for initial project scaffolding.
Info
You can find the source code of the template in this repository.
TL;DR
To use the template just run:
Before going further with the implementation details, I want to clearly define the scope of the template I'm sharing.
Goals and requirements
I didn't have a goal of creating an all-encompassing template to cover every possible use case. This template is heavily opinionated and tailored to my needs. But I believe you can adapt it to your needs without much effort.
Below are the assumptions I kept in mind while creating the template.
Functional requirements
- The project intended use is fairly simple network automation scripting.
- It must be consumed as a CLI tool (Typer and Rich).
- It can be run either as a Python script or as a Docker container (see my previous post).
- It can be run either locally (when developing) or as a CI/CD job on a self-hosted Gitlab CI instance.
- It's supposed to connect to network devices and query the Source of Truth system, in my case Nautobot (Nornir and Netmiko).
- Because it interacts with external data sources, data must be parsed and validated (Pydantic).
Code management requirements
- Packaging and dependency management must be automated (Poetry).
- The resulting project must adhere to the src layout.
- Common housekeeping tasks must be automated (Invoke).
- The code must be type-hinted and checked with mypy.
- Linting and formatting must be applied to code before committing to Git (pre-commit).
- Commit messages must follow the Conventional Commits specification (Commitizen).
- Changelog must be created automatically and follow the Keep a Changelog specification. Also covered by Commitizen.
- There should be a choice of license when creating a project. Currently, I included only APL 2.0, GPLv3, and MIT. Easily extendable.
What was left out of the scope
At least for now, this template doesn't include:
- Documentation building tools
- Test automation tools
CODE_OF_CONDUCT.mdCONTRIBUTING.md
Choice of tooling
A quick googling for project templating software gives not so many options.
This is the list I've been stumbling upon here and there:
- Cookiecutter. Probably the most popular one, with lots of pre-made templates. Written in Python.
- Yeoman. A bit less popular, and focused on web apps. Written in JS.
- Copier. The least popular, but modern and has some unique features. Written in Python.
I chose Copier because of my irrational desire to try everything new and shiny. The ability to apply updated templates to already created projects also had its appeal.
The project's documentation features a nice comparison table where you can find key differences from other templating tools.
Project structure
Now that we know what we want let's look closer at the template implementation.
Below is the project structure of the template. Click for more details.
.
├── project
│ ├── docker
│ │ ├── Dockerfile.jinja
│ │ └── docker-entrypoint.sh.jinja
│ ├── src
│ │ └── {{python_package_import_name}}
│ │ ├── __init__.py.jinja
│ │ ├── cli.py.jinja
│ │ ├── data_models.py.jinja
│ │ ├── logger.py.jinja
│ │ ├── py.typed
│ │ ├── run.py.jinja
│ │ └── settings.py.jinja
│ ├── tests
│ │ └── test_version.py.jinja
│ ├── .gitignore
│ ├── .gitlab-ci.yml.jinja
│ ├── .pre-commit-config.yaml.jinja
│ ├── LICENSE.md.jinja
│ ├── README.md.jinja
│ ├── pyproject.toml.jinja
│ ├── run-in-docker.sh
│ ├── tasks.py.jinja
│ └── {{_copier_conf.answers_file}}.jinja
├── .cz.toml
├── .gitignore
├── .pre-commit-config.yaml
├── CHANGELOG.md
├── LICENSE.md
├── README.md
├── copier.yml
└── tasks.py
Info
A few words on how Copier works. On a basic level, it's pretty straightforward: a user runs copier command, answers configured questions, Copier renders the target files from Jinja2 templates, substituting the variables with values provided by a user.
This layout differs a bit from what Copier expects by default by placing the template in a separate directory called project. This enables a clear separation of the template and its metadata.
Gitlab CI requirements
The .gitlab-ci.yml configuration supplied with this template expects the following from your self-hosted Gitlab CI instance:
- A runner with the Docker executor tagged with
docker. - A runner with the Linux shell executor tagged with
shell. - Because the
shellrunner is used to run your automation script it is supposed to have network access to the external resources your script interacts with (e.g., ssh access to network devices).
The following environment variables are supposed to be set and accessible by your Gitlab project:
| Variable | Description |
|---|---|
PROJECT_ACCESS_TOKEN |
Project access token with read-write access to the repository and the ability to push to the main branch |
CI_USERNAME |
Arbitrary git username for the version-bump CI job, e.g. ci-bot |
CI_EMAIL |
Email address of the CI_USERNAME, e.g. ci-bot@example.com |
DOCKER_REGISTRY_RO_PASSWORD |
Account password with read-only access to the private Docker registry |
DOCKER_REGISTRY_RO_USER |
Account username with read-only access to the private Docker registry |
DOCKER_REGISTRY_RW_PASSWORD |
Account password with read-write access to the private Docker registry |
DOCKER_REGISTRY_RW_USER |
Account username with read-write access to the private Docker registry |
DOCKER_REGISTRY_URL |
Private Docker registry URL |
Of course, you are free to adjust anything to your needs. Just make sure that the environment variables used in the .gitlab-ci.yml match those configured in Gitlab.
Demo
Now let's generate a project out of this template.
🎤 Maximum line length (used by linters and formatters) 99 Copying from template version 0.1.1 create . create docker create docker/Dockerfile create docker/docker-entrypoint.sh create README.md create tests create tests/test_version.py create .copier-answers.yml create .gitignore create .gitlab-ci.yml create tasks.py create LICENSE.md create .pre-commit-config.yaml create pyproject.toml create run-in-docker.sh create src create src/awesome_automation_project create src/awesome_automation_project/cli.py create src/awesome_automation_project/data_models.py create src/awesome_automation_project/run.py create src/awesome_automation_project/__init__.py create src/awesome_automation_project/logger.py create src/awesome_automation_project/py.typed create src/awesome_automation_project/settings.py ~ took 34s ❯
This created the project directory with all the needed files.
Now we can install the dependencies and check that the hello_world function from the included script works.
~ ❯ cd my-project ~/my-project is 📦 v0.1.0 via 🐍 v3.11.3 ❯ git init Initialized empty Git repository in /Users/dteslya/my-project/.git/ my-project on main [?] is 📦 v0.1.0 via 🐍 v3.11.3 ❯ invoke install Creating virtualenv awesome-automation-project in /Users/dteslya/my-project/.venv Updating dependencies Resolving dependencies... Writing lock file Package operations: 101 installs, 1 update, 0 removals • Installing pycparser (2.21) • Installing cffi (1.15.1) • Installing bcrypt (4.0.1) • Installing cryptography (40.0.2) • Installing future (0.18.3) • Installing pynacl (1.5.0) • Installing six (1.16.0) • Installing lxml (4.9.2) • Installing markupsafe (2.1.2) • Installing paramiko (3.1.0) • Installing pyyaml (6.0) • Updating setuptools (67.6.1 -> 67.7.2) • Installing textfsm (1.1.2)
Warning
I deliberately included a couple of dummy secrets in settings.py file just for the demo to work out of the box. Otherwise, I'd have to define them as environment variables before running the demo. Don't forget to remove them before committing your changes.
At this point, you're ready to proceed with the development of your script.
Acknowledgments
- How to use copier to create project templates by Haseeb Majid
- Copier Poetry template by Timothée Mazzucotelli
- Python Packages Project Generator by Roman Tezikov