Getting Started

Getting Started#

Creating repositories#

First setup#

If you’re starting a new repository, go to that repository and then run copier copy --trust gl:znicholls/copier-notebook-based-book path/to/destination/repository. Copier will ask you the required questions and then setup your repository for you.

Alternatively, if you have the template repository cloned locally, the path to the repository can be used instead of gl:znicholls/copier-notebook-based-book.

Copier will use the latest tagged release for generating a new project. If you wish to use a specific commit/tag the --vcs-ref flag can be used (--vcs-ref HEAD will use the most recent commit).

To run this step, you will need to have copier installed globally (we can’t make the virtual environment first as that is done with the Makefile, which copier creates for us).

Once you have created your repository, there are then a number of further steps which have to be done to get everything running as intended. The sections below go through these steps.

GitLab#

If you’re hosting the repository on GitLab, do the steps in this section too. Otherwise, skip this section.

In the GitLab page for your project, under Settings -> CI/CD, go to the “Variables” heading.

Add the known GitLab host keys (these can be generated with e.g. ssh-keyscan gitlab.com) to a variable called SSH_KNOWN_HOSTS. Make it of file type and protected (so it can only be accessed in protected branches and tags) but don’t worry about masking it.

Now, create an ssh key without a passphrase. This can be done with e.g. ssh-keygen -f deploy_key -N "". You need to put the private key in a GitLab variable called SSH_PRIVATE_KEY. Make it of type “File” so that it can effectively be masked by GitLab. Make sure it ends with an empty line so that it is properly parsed later on. Once again, make it a protected variable.

Next, add the public key as a deploy key to your project in Settings -> Repository -> Deploy keys. Grant write permissions to the key and set an expiry date if you want (after which time you’ll need to rotate the keys).

Now, go to Settings -> Repository -> Protected tags and add “v*” as a protected tag. This will ensure that commits created with bump can also access the variables we are setting up. By default, the main branch should be protected so you shouldn’t need to change the protected branches at all (but feel free to check that main is actually protected and make it protected if main isn’t already protected for some reason).

If the repository has a protected main branch which disallows anyone from pushing, the newly added deploy key must be added to the “Allowed to push and merge” user set. If you don’t do this you will get an error stating “You are not allowed to push code to protected branches on this project.”. If that happens, contact your administrator as the CI may be in an odd state that requires cleaning the existing docker containers.

If you wish, also add a pipeline schedule to run the scheduled jobs on a schedule of your choosing. These jobs do automated maintenance checks, like checking that the book builds from scratch.

Checklist:

  • [ ] SSH_PRIVATE_KEY CI variable containing private part of the SSH key

  • [ ] SSH_KNOWN_HOSTS CI variable containing the result of ssh-keyscan gitlab.com

  • [ ] Deploy key with write permissions using the public part of the SSH key

  • [ ] Protect the “v*” git tag

  • [ ] Add a pipeline schedule for automated maintenance jobs

This section is based on the the commitizen docs.

GitHub#

If you’re hosting the repository on GitHub, do the steps in this section too. Otherwise, skip this section.

Create a personal access token that can be used to write to the repository as part of GitHub actions. It is best to use the fine-grained tokens and only give the token access to this repository. The token will need “Contents” permissions, specifically read and write access for “Contents”.

If you can’t create a token, your organisation may need to enable personal access token access. Please ask one of the lead developers for your organisation.

Once you have your token, add it to a repository secret (Settings -> Secrets and variables -> Actions -> New repository secret) called PERSONAL_ACCESS_TOKEN.

Checklist:

  • [ ] PERSONAL_ACCESS_TOKEN repository secrete containing a personal access token (for a single repository only) with read and write access for “Contents”

Pre-commit CI#

On GitHub, we can make use of pre-commit CI to run our pre-commit jobs (effectively our linting). This is configured by our .pre-commit-config.yaml file. The only step required to get it to run is to ensure that pre-commit CI is configured so that it has access to this repository. To do this, go to installations in your organisation’s GitHub organisation page (the URL will be something like https://github.com/organizations/my-org-name/settings/installations, alternately you can get there by following https://github.com/organizations/my-org-name -> Settings -> GitHub Apps) then press ‘configure’ next to pre-commit CI. Ensure that pre-commit CI has access to this repository or add access, then you’re all setup. On each pull request, the pre-commit tasks will be run on all files and any failures will be reported in the pull request page.

Read the Docs#

At the moment, we only integrate with readthedocs.org hence this setup only works for public repositories. If you want to host docs for a private repository, you will need to get setup with Read the Docs for Business.

If you haven’t already, you will need to link your GitHub/GitLab account with RtD. This can be done by (if you don’t already have one) creating an account with RtD and then going to Settings -> Connected services. From here, the connection can be trivially added by pressing “Connect to GitLab/GitHub” and following the prompts.

To add your project, go to your RtD dashboard then press “Import a Project”. Pick your project from the list then simply follow the setup instructions (you can mostly just leave the defaults as they are, the only thing we have done in the past is adding tags).

Once your project is working with RtD, we recommend going to Admin, scrolling down/searching and ticking the “Build pull requests for this project” box. That will cause the docs to be built on all pull requests, which gives fast feedback about whether a change will a) break the docs and b) cause the docs to be updated in the expected way.

It is also worth going to Versions and making sure that the “latest” (generally the last commit merged to main) and “stable” versions are both active so that users can see docs for both these versions.

Updating repositories#

If you need to update your repository, simply navigate to your repository and run copier update. If you don’t want to go through all the questions again (the default answers are taken from last time you answered the questions), use copier update --force instead.

By default, copier will use the most recent tag when updating the repository. If you wish to use the current HEAD commit for your update, run copier update --vcs-ref=HEAD. This --vcs-ref option can also be used to specify a specific tag to apply.

When you update, there will likely be merge conflicts, particularly in pyproject.toml related to versions. If you use the --conflict inline option with copier update then the diffs should be inline (this is apparently experimental though, see here). These diffs can then be resolved manually. The pre-commit config will make sure you don’t miss conflicts and accidentally commit merge conflict lines.

Any conflicts related to the poetry.lock file can be safety ignored and a poetry lock should be run after updating to regenerate the lockfile.

Contents