ASWF CI Working Group

Meeting: 5 February 2020

Attendees

• Daniel Heckenberg (Animal Logic, TAC Chair)
• Jean-Francois Panisset (VES Technology Committee)
• Aloys Baillet (Animal Logic)
• Jeff Bradley (Dreamworks)
• Michael Dolan (OpenColorIO)
• Dan Bailey (ILM / OpenVDB)
• Christina Tempelaar-Lietz (OpenEXR)
• Larry Gritz (SPI)
• Andrew Grimberg (Linux Foundation Release Engineering)
• Doug Walker (OCI / Autodesk)
• Trevor Thomson (Blue Sky)

Agenda & Notes

ASWF CI Goals for Year 2

Will propose and discuss and ordering of tasks at next week TAC’s meeting (Daniel).

• GPU
  • GPU build and test as normal part of CI infrastructure, not just a PoC
• Python3 / VFXPlatform 2020
  • State of Python 3 in OSS and commercial apps: https://docs.google.com/spreadsheets/d/10XG92byepTD-LEeXx4mBjhGaNPtJsd_QaXlZ866wj7k/edit#gid=0
  • Jeff: what’s the timeframe for a Python 3 working group? Daniel: will be defined at next TAC meeting (also USD working group).
• Windows, Mac
  • VFX Reference Platform to target Windows, Mac, specifically compiler specifications
• Runtime tests with commercial components
  • Limited examples so far
  • Animal Logic has been working on Maya USD plugin with other studios and Autodesk, need to test with Maya
• Packaging / signing
  • Daniel: touched on this a few times, smaller studios would like to have binaries they can consume.
  • Would this help for developer engagement and outreach?
  • Or is this a service for end users?
  • Client OSes are ratcheting up the signing requirements for binaries: OpenEXR tools, OpenCue applications.
  • Studios run software downloaded and built without too many tools and guarantees of authenticity.
  • ASWF can own certificates / signing authority. We can provide a secure and authenticated software delivery path.
  • Dan: GitHub Releases would be a good place to distribute binaries. Further incentive to automate releases. 2GB file limit.
  • Dan: also getting involved with vcpkg and other package managers so users can get our software throught those packages.
  • Michael: recent requests for vcpkg packaging of OpenColorIO
• GitHub / Azure Permissions
  • GitHub Actions should make it easier to manage CI aspect
  • Daniel: asking Dan whether this actually helps? Dan: a user without write permissions doesn’t see the Actions tab on the repo. If you have write but not admin permissions you can see the steps happen. So priority would be to get some permissions on GitHub.
  • Andrew: right now there are no environment variables to share credentials between repos in a GitHub. Could be very useful for sharing SonarCloud credentials for instance. Should hopefully be available from GitHub Actions soon at the Organization level, with whitelisting of which repos can access credentials / secrets.
  • For now these “shared” credentials would have to be replicated in every repo for GitHub Actions.
  • Dan: what about making tweaks to a repo, still need to get the releng team involved (service desk ticket). Andrew: should have TAC level discussion as to allowing some TSC committers repo admin privileges. Concern is bypass of security related processes.
  • Andrew: there is auditing available, but the GitHub audit logs tend to be massive, so they are not monitored on a recurring basis.
  • Dan: what can we do to make the releng team comfortable? Andrew: some of the ASWF projects already have that, so some users with committer rights also got admin rights. If we’re going to go down that road, we need to provide an explicit list of who has / needs what rights. We need to make sure we don’t compromise the security posture we are agreeing to. Rights management is not as structured with GitHub as it is with Gerrit (used by some other LF projects).
  • Andrew: there is now a “maintainer” role at the repo level, every right an admin has with some very specific exceptions, so we should look at role. Admins can still sidestep some requirements since they can turn off “branch protection” for instance, maintainers cannot. For instance don’t want to be able to turn off CLA checks.
  • Daniel: asking Dan whether “maintainer” would satisfy requirements? Dan: will look at it.
  • Dan: A document in the repo that lists who has what rights could be useful. Andrew: yes that would help. Dan: will go ahead and put something together, Andrew to review.

• Exploring GitHub Actions

  • Daniel: some explorations
  • Dan: have been running Azure Pipelines and GitHub Actions in tandem, looking to switching to GitHub actions as main CI
  • Some hiccups that need to be resolved.
  • GitHub Actions gives you separate files per workflow, want to automate the release process, and GitHub Actions seems to suppor tthat well.
  • Azure: pull requests from forks don’t have access to any of the secrets, you can flag secrets to be shared, but that breaks the security model. Little better than storing passwords in source files.
  • Could change standard practice to having developers create PRs against a devel branch, and TSC members would be responsible for then merging into the master branch. Is it a good idea to modify someone else’s PR? But VDB has stringent requirements before something can be merged. But that can help to lower the barrier to contribution, and can make it easier to share secrets with a specific devel branch.
  • GitHub Actions is felt to be less complex than Azure Pipelines.
  • Is OpenVDB the only project with secrets (Houdini download)? Aswf-docker project has secrets for pushing to Docker Hub repository. Also shared secret between all projects for SonarCloud.
  • Forks need to have access to the same testing infrastructure, so we definitely need a solution for that which allows developers access to the infrastructure we have in place.
  • Should we try to document / script the infrastructure our projects depend on so that individual contributors can stand up their own infrastructure.
  • Larry: now using GitHub Actions as primary CI for OIIO and OSL, still running others in tandem, only reason Travis is still running is due to ease of using older compilers, but can be solved with Docker. Seems to improve over time, and happy with GitHub Actions.
  • Daniel: some projects will want to move faster to a new CI infrastructure, some will want to stick with what we currently have.
  • Andrew: Microsoft strongly pushing towards GitHub Actions for Open Source projects in call with LF releng team, more resources allocated to it.
  • Larry: haven’t observed GPU support in GitHub Actions, but OSL is starting to need to test on GPU.
  • Andrew: same problem on GitHub Actions than on Azure Pipelines, only way for now is to add a builder to a static instance pool. GitHub Actions team had promised some kind of dynamic instance creation by end of quarter.
  • Michael: Larry’s proposal for OSL would help to justify GPU infrastructure, would no longer just be OCIO.
  • Aloys: will same Docker images work across Azure and GitHub Actions? Dan: yes, seems to work fine.
  • Larry: is this checked in to current OpenVDB repo so we can look at GitHub Actions scripts? Dan: yes, Linux only for now, but has been in the repo since October / November 2019.
  • Daniel: sounds like we have a strong agreement to migrate to GitHub Actions. Will be brought up at next week’s TAC meeting and come up with a timetable.
  • Dan: we should agree to not change our CI for another year!

GPU Resources

• Static / dynamic provisioning
  • We need a concrete path to solve the dynamic provisioning challenge
  • Two options: wait for functionality from CI providers (Azure Pipelines and GitHub Actions promising something in next few months)
  • Or spinning up static instances: pushback from LF Sysop group due to billing problems
  • Or manual / on demand provisioning for specific build / test steps
  • Andrew: LF has a master AWS account, and sub accounts dependent on that. So all billing gets invoiced to LF, but broken down by project. If we are doing dynamic instances, each project could be accounted against their account. AWS credits are only applied against the top level account, not the project-specific account. And can only get invoiced (rather than credit card based) if you have enough traffic, so couldn’t have a ASWF-specific top level account.
  • Andrew: static, long lived build instance falls under security, auditing requirements, all of their LF systems are short lived, and LF sysop doesn’t have the infrastructure to deal with that.
  • LF releng rebuilds the Jenkins instances every month on the OpenStack infrastructure.
  • JF: need to make sure for-pay GPU instance cannot be abused, perhaps GPU testing can only be launched from master branch than from a PR in a separate branch? Also should target GitHub Actions if that’s the direction we are going. Will need some help with how to tie this into project workflows.

Project Specific Goals

General Discussion

• Aloys: aswf-docker now includes USD 1911, so can be used to run tests. VFX Reference Platform currently doesn’t specify a USD version, so not super clear which version to target.
• Aloys: should we still maintain Python 2.7 support in VFX Platform 2020 build containers? Makes everything more complex to maintain 2 Python versions.
• Christina: OpenEXR was trying to run both Python 2.7 and 3, but it’s fine to keep separate containers, run Python 2.7 in the VFX 2019 container, and 3.7 in the VFX 2020 container.
• Daniel: ASWF survey showed that everyone seemed quite pleased with the work delivered on the CI side.

Next Steps

• Follow up meeting: 4 March 2020

---