*SHAMELESS PLUG*
I’ll be talking about this piece of our infrastructure and more at my PSU Mac Admins conference talk on July 14th (video will be posted here too as soon as it is available).
For a while now, we’ve put our Munki repo in Git with git lfs. This has been a fantastic boon as we work to distribute our tools across a wider range of offices, but also came with a significant downside: AutoPkgr, which we’ve been leveraging for a while to find and import new updates, is completely unaware of git. My attempts to make it at least add things and commit / push were feeble, and I couldn’t quite get the recipe order to always work out.
Our developers have long used GitLab, and in the last few months I’ve been relying more heavily on the GitLab CI runners to accomplish routine tasks and automate my internal deployments. As our IT team expands, we need to ensure that we have a defined and logged process that we use to deploy our software – using GitLab CI allows us great transparency.
A few weeks ago, GitLab finally released the ability to schedule CI runs in a cron-like format. I instantly got to work thinking about a wrapper script to make this work. I’ve settled on the below process for now, but I’m sure the community has suggestions on a better way.
AutoPkg Wrapper Script
The least exciting part of this was the wrapper script, mostly because Facebook’s incredible CPE team had done the overwhelming majority of the work for me! I modified their autopkg_tools.py script with a few new functions and behaviors to better fit our git flow process and the frequency with which we release updates. I also filled in some glue between our Trello instance and the script so that we get a new card added whenever software is imported for us.
You can read through it on the gist link below – I’m not going to embed it because it is 600+ lines of code!
https://gist.github.com/rickheil/128a282e6301fc43c5a4e567554ed242
CI Setup
GitLab CI is radically different from Jenkins (the other CI I have experience with) in that very little configuration of the CI is done by logging into GitLab in your browser – the entire process is defined in the .gitlab-ci.yml file in your repository. This makes our CI process easy to track and update, and continues the “infrastructure as code” push we are making.
GitLab provides a function to set values as environmental variables – this helps us keep things like API keys and tokens out of the git repo. Set them in Settings -> CI/CD Pipelines – ours look like this:
You’ll also want to create a trigger for that TRIGGER_TOKEN value (just underneath where you set your variables on that same page). I just named mine “Periodic CI.” You can then use Cron syntax to schedule when your CI job will kick off. See the example below to trigger every four hours.
Finally, the YAML we are using is below. It’s very simple – an “OS X” tags ensure that the job runs on one of our macOS runners (a few Mac Minis), and the script only executes on the master branch.
How it Runs
Every four hours (or on manual creation), a CI pipeline kicks off on the master branch. The $RECIPE_NAME variable is not passed by cron or manual kickoff – and because that value is empty, the run-autopkg-recipes.sh script is kicked off. That simply iterates over the recipe overrides in the overrides/ directory of the repository, and creates a cURL call to GitLab for each one – creating a new pipeline for each recipe we run.
This first CI job creates logs that look like this:
Having the status code can be useful to figure out if/why a job didn’t trigger. Since the run-autopkg-recipes.sh script pushes the RECIPE_NAME variable, those new CI scripts call the autopkg_tools.py script to run that specific recipe. The Trello information is passed from the GitLab CI variables.
As those single-recipe pipelines run, they create a new feature branch using git flow and the exact name and version of the new software to be imported. At the end of the run, it pushes that feature branch up to the munki git repo and adds a card into Trello to notify us something is ready to test!
Final Thoughts
Having a fully cloud-based Munki setup has been a goal of mine for a while – and now, the only remaining on-prem piece are the GitLab CI runners for AutoPkg. As we think about expanding our management platform beyond one or two engineers who sit next to each other, having the scalability, transparency, and flexibility of our infrastructure being run by CI in the cloud has been a boon. It takes a lot of the guesswork out of management and means we don’t have to worry about various VPNs, site to site links, and more that might usually plague mergers.
Again – I’ll be talking about this piece of our infrastructure and more at my PSU Mac Admins conference talk on July 14th (video will be posted here too as soon as it is available).