3. Development Workflow

After the development of a specific ticket(task) locally, the work is first reviewed and when approved it is deployed to the staging server. Afterwards, if the task is approved by the Pearl Team, it is marked as done and will be deployed as part of the current release to the production server.

Our development workflow is release-based. In order to support this flow we use mainly Jira and GitFlow.

3.1. Jira Workflow

Jira is the project management and issue tracker that we use. It enables us to define our development workflow as well as track issues across their different statuses. You can check the latest release in the Kanban board here.

There are 2 Jira Instances for managing the Pearl project: Pearl Jira and SC Jira.

Data (tickets, comments, transitions, …) is synced between the 2 instances using the “Backbone Issue Sync” plugin.

Let’s take a quick look into how work is divided between the two Jiras. First, the tickets are created and specified on the Pearl Jira as part of a release. Once they are “Ready”, they are assigned to SC to start work on them. Tickets are, of course, synced to SC Jira and are divided between team members who start work on them. After work is finished, the tickets are deployed to the staging server and the Pearl Team is notified to test them (mainly transitioning the ticket to the “Client Testing” status).

3.1.1. SC Jira Workflow Overview:

  • Backlog: all newly created tickets start in this status.

  • Selected for Development: Once tickets are fully specified, they go here. Also, they are assigned to a Dev Team member so he start working on them.

  • In Progress: When the assignee starts work on a ticket, he should move it to status so that the ticket status is known by everyone.

  • SC Test & Review: The ticket should be changed to this status after work on it is completed so that a reviewer can check it and point any issues (if any).

  • Client Test: The ticket should be moved to this status only after it was approved by the reviewer and deployed to the staging server.

  • Done: The ticket will be moved to the this status only after the Pearl Team has approved it and that means it is ready for deployment to the production server.

3.2. GitFlow:

As the work is release-based, we chose to use the GitFlow branching model.

3.2.1. GitFlow Overview:

As the work is release-based, we chose to use the GitFlow branching model.

You can find a quick overview here, and a more detailed explanation here.

Basically, work on a ticket should be done in a new feature (in the GitFlow sense). The name of a new feature should be prefixed with the ticket key and the the ticket name. An example of a feature name would be “PCS-482_new_greenfield_report_for_long_island”. After work on a feature is done it is merged back to the “develop” branch. “develop” is the branch that the staging server pulls from. Once all tickets in a certain release are done and merged, a release branch is started and then merged back to “master” and “develop”. If there are any bugs in production, a hotfix branch should be created to fix the issue.

We use “sourcetree” as a git GUI that has the GitFlow integration by default to ease the use of the GitFlow branching model within the project.

3.2.2. Gitflow & Pull Requests:

Gitflow defines 3 types of branches: features, releases and hotfixes.

In their simplest form, pull requests are a mechanism for a developer to notify team members that they have completed a feature. You can also find good general advices about pull requests here.

Now let’s see how to use Gitflow and Pull requests together to get the most out of them.

Features:

  • Naming: Names of feature branches should start with the ticket key and then the ticket title. Example: “PCS-470 Add email notification to contractors”.

  • Pull Request: the pull request should be opened against the “develop” branch because the feature when finished will be merged to develop.

Releases:

  • Naming: Names of release branches should be the version name for that release on Jira. Example: “v1.4.6”.

  • Pull Request: the pull request should be opened against the “develop” branch because the release base branch is develop.The main use for the PR in this case is to make sure that the branch was merged.

  • Notes: make sure to add a tag for the release and then push it to github. A tag helps in quickly checking out a certain release.

Hotfixes:

  • Naming: Names of hotfix branches should follow this convention “<version_name>”. Example: “v3.52.1”.

  • add notes to prod_notes.dev. The format should follow the existing examples

  • update the CODE_VERSION variable by updating the patch number. If current version is 3.52.0 then the first hotfix after it is 3.52.1

  • Pull Request: the pull request should be opened against the “master” branch because the hotfix base branch is master. The main use for the PR in this case is to make sure that the hotfix was merged.

  • Notes: make sure to add a tag (e.g. v3.52.1) for the hotfix and then push it to github (the deployment requires the existing of the tag). The tag should be on the commit that marks the merge of the hotfix to master.

3.2.3. Ticket (or task) Lifecycle:

Once a ticket is fully specified and ready for work, the first thing to do is estimate the time needed to finish this ticket. This estimation should be based on the changes that need to happen in order to finish the task. Now that the changes that need to happen are known, a new feature should be opened to work on this ticket (feature name should be prefixed with Jira issue key). When starting a new feature, you need to pull latest from develop, check the notes in “notes/stage_notes.md” and apply the instructions after the last note you applied before (still in some cases one might need a fresh stage db dump). Besides, the ticket should be moved from “Selected for Development” to “In Progress”.

After finishing the work on the ticket, make sure:

  • To read ticket description again to see if you missed any requested changes, AND

  • To test the changes from scratch especially if there db migrations involved (with a new db generated from the develop branch)

  • To add notes about the ticket (migrations, scripts that need to be run, …) to the “notes/dev_notes.md” file (follow already used format).

After testing the ticket, move it from “In Progress” to “SC Test & Review” and create a pull request on github against “develop” (NOT “master”).

Afterwards, a reviewer will test and review the task and add any comments to the pull request on github. If there are any comments that need more work, the ticket will be moved from “SC Test & Review” to “Selected for Development” so that the assignee can fix the mentioned issues. Once the ticket is approved by the reviewer for merge, it is deployed with other tickets on the staging server.

If the Pearl Team decides that a ticket still has work to be done (missing feature, misunderstood specs …), the new changes should be done in a new feature branch (and NOT the old one as it was already merged) with the same feature naming convention with a small addition: <issue_key>-feedback-<issue_title> so it can be distinguished from the original feature. If the requested changes is very simple (spelling mistake, reword sentence …), it can be pushed directly to “develop” but make sure that the commit message is prefixed with the issue key.

When the Pearl Team approves the ticket, it is moved to the “Done” status and that means it’s resolved. If an issue related to that ticket is discovered later (in the next release for e.g), a new ticket will be opened to handle the issue.

3.2.4. About Commits:

While working on a feature, make sure to prefix commit messages with the issue key (e.g. PCS-482). That way commits linked to the issue appear in the corresponding Jira ticket (on the right bottom). Also, and in general, make sure to separate your commits by logical context as it is not recommended to make large commits (each with multiple files) as this will make it harder for the reviewer to do his job, and no need for fragmented commits (with one line each) as there is rarely any need especially when working on new features. Of course valid exceptions exist in both cases, you can still commit multiple files if you’re, for example, adding a new js plugin to staticfiles and, one line commits may be necessary when fixing bugs or making code optimizations. You can also find some commit best practices here.

3.2.5. Checklist for merging features to develop:

Merging a feature to develop is done by the reviewer of the feature.

Prerequisite:

  • The developer working on the feature should always merge develop into their feature before requesting a review to avoid missing issues that only appear when merging to develop.

The following are steps to follow when merging a feature to develop:

  • The reviewer should first merge develop into the feature and resolve any conflicts (if any) since there can be changes between develop merge by the developer and develop merge by the reviewer.

  • check notes/dev_notes.md to see if there are migrations for the feature.

  • If there are migrations, remove any migrations created in the feature and regenerate them so we always add the least number of migrations to develop.

  • if there are no migrations pushed, make sure to run python manage.py makemigrations to check if the developer missed pushing them. This needs to be done for all sites (cert, green_door, pqs, api)

  • make sure that python manage.py migrate works with no issues on a fresh db based on the latest develop. This needs to be done for all sites (cert, green_door, pqs, api)

  • set the env var ACTIVATE_WHITENOISE=True and run python manage.py collectstatic and make sure that it does not fail. This needs to be done for all sites

  • check if there are any scripts created in the PR but were not added to pearlcertification.core.management.commands.runscripts, that the release version matches the current release and that the site is set properly for green door scripts

  • check if there was any new settings with env variables added and make sure that there is a note in notes/dev_notes.md even if a default value is set.

  • check if there are any new python/django packages in the feature that were not added to requirements/base.txt and requirements.txt, if so move the ticket back to “Selected for development” and notify the developer about this issue.

  • move notes from notes/dev_notes.md to notes/stage_notes.md. If there are no notes, make sure to sure still add a note to notes/stage_notes.md with the text “no migrations” (notes format should match the format of previous notes).

  • Now the feature can be merged, checkout the base branch and select “Merge into Current” under the feature, if the base branch is develop we can simply do Gitflow > Finish feature.