Link Search Menu Expand Document

ASWF CI Working Group

Meeting: 18 September 2019

Attendees

  • Daniel Heckenberg (Animal Logic, TAC Chair)
  • Trevor Thomson (Blue Sky)
  • Aloys Baillet (Animal Logic)
  • Brian Cipriano (Google / OpenCue)
  • Michael Dolan (SPI / OpenColorIO)
  • Charles Bouet
  • Jean-Francois Panisset (VES Technology Committee)
  • Larry Gritz (SPI)
  • Dan Bailey (ILM)
  • Gordon Bradley (Autodesk)
  • Sean Looper (AWS)

Agenda & Notes

ASWF CI Goals for Year 2 [0:00-0:10]

Outstanding / Ongoing:

  • Will realign ASWF goals towards calendar year in 2020
  • CI Documentation, Guidelines (JF) Just started, will have candidate for becoming official ASWF repo by next month https://github.com/jfpanisset/aswf_sample_project
  • Dependency management
    • Discussing with Allan Johns, potential of bringing Res as official ASWF project
  • GPU
    • Andrew: LF cost-managed Azure GPU pools
    • Sean: AWS provided GPU instances. Need to get AWS account number for ASWF, and estimate of how much resources are required / estimated cost. Should be spun up on demand rather than running all the time. We don’t need the low latency of an “always on” GPU instance.
  • Commercial components
    • Need to expand our base of software we can use at build / test time.
    • Gordon: Autodesk can donate licenses to ASWF to run tests, but where do we store the keys? Daniel: OpenVDB is using a stored key for Houdini (unclear how you share a key across a forked repository). No runtime testing for now, but should be runable with a Apprentice license.
    • Sean: AWS has usage based licensing for applications, so could also use that mechanism to use commercial licenses for testing (metered licensing).
    • Gordon: seems like we have some of the components in place, but unclear how it all fits together so we can run a test on a GPU instance that accesses licensed software. There needs to be an actual project to demonstrate how this would work.
    • Daniel: is there a project that already has Maya based tests? Dan: building against Maya SDK would be the first step.
    • Gordon: not clear who would review / approve the process. The ASWF will have to own some of the responsibility, will need to put a proposal to Autodesk. Sean: AWS has the ability to purchasing “hours” of usage ahead of time. Daniel: we might want to run these tests on other infrastructure than AWS, but this could help with the current, fairly narrow use case. Sean: Azure and GCP also provide this, may be simplest way to get to where we want to be. Gordon: is this already hooked up to Autodesk licenses? Sean: yes, mainly for rendering. Gordon: this could be the path of least resistance, but eventually would be preferable if Autodesk would donate licenses rather than charging.

Follow ups: [0:10-0:20]

  • Investigate GitHub Actions CI?
    • Allan Johns / rez accepted into Beta program and investigating.
    • JF: GitHub Actions: no current support for self-provided build instances (so no GPU support for now). Uses Azure Standard_DS2_v2 instances for Windows/Linux and MacStadium for macOS: https://help.github.com/en/articles/virtual-environments-for-github-actions
    • LG: has set it up for OIIO and now has some experience. Playing around with it for a couple of weeks, working for Linux and Mac, working on Windows (building, not tests yet). Seems equivalent to other CI systems, should offer nice integrated experience on GitHub. No run to run caching (so longer builds - no caching dependencies), missing polish for now. Would not recommend switching to it from another system right away, but seems promising. Interesting to see where it is in a year or 2.
    • Dan: permission model seems much simpler than Azure Pipelines.
    • JF: GitHub Actions agent is a fork of the Azure Pipelines agent.
    • Daniel: hopefully we can get some insight from our new Microsoft representative.
    • Beta now has instant acceptance (as of this morning)
    • Dan: just signed up for GitHub Actions, was able to copy over and adapt the existing azure_pipelines.yaml file to get a simple build going (most of the complexity is in the build scripts). Argues for keeping complexity out of the CI-specific files.
  • Standardization of additional dependencies?
    • OpenCue: JDK (Brian / Aloys)
    • Aloys: fixed up Docker images so that you can run “yum” and install the Java JDK
    • Brian: built CI Docker image for OpenCue, no big rush to add to standard images since no other project currently uses JDK.
    • Switched OpenCue to published Docker image, working on SonarCloud, improving unit test coverage. Coverage numbers are going up.

CI Updates for Projects [0:20-0:40]

  • OpenEXR
  • OpenCue
  • OpenColorIO
    • Michael: ran into Azure Pipelines issue, when AP reports name of jobs back to GitHub for PR approval, started changing the casing of the job name, which broke PR approval, had to approve PRs manually since they weren’t showing as having passed the build. Working with Microsoft tech support to fix (Static converted to static in job name).
  • OpenTimeLineIO
  • OpenVDB

CI Platform [0:40-0:50]

  • Staging, Organisations, Roles, Access
  • Managing “latest tag” / top of tree builds
    • Michael: responded on email thread with examples of how OCIO is doing that. Dan: do we want to support this through Docker? MD: OCIO has a “latest” build that runs in cron, installs latest builds from dependant repos to catch breakage with new versions. https://lists.aswf.io/g/tac/message/877
    • Aloys: started to add tests at end of Docker build for CI images, run a build from OpenVDB to make sure new Docker images don’t break builds. Should there be test builds from other projects such as OpenColorIO? MD: can help with that, fairly easy / fast to do a minimal OCIO build.
  • Support for Python3 transition
    • Is our CI infrastructure in a good position to support Python 3?
    • What can we do to help with the transition?
    • Effective use of test and CI to ensure Python 2 and 3 compatibility (discussion at SIGGRAPH)
    • LG: do all of our current adopted projects support Python 3? Should we clean up our own house first? Do our C++ libraries with Python bindings are Python 3 compatible? Dan: PR went in for Python 3 VFX Reference Platform 2020 container.
    • Aloys: OpenEXR ilmBase does not compile with Python 3. But CI base image with Python 3 (with boost-python) is built.
    • LG: OpenEXR should be days away with a 2.4 release, after that will switch Python bindings to be Python 3 compatible.
    • MD: OCIO working on it as well.
    • BC: OpenCue, 3 of 4 components are Python 3 compatible, CI setup can pick 2019 or 2020 Reference Platform image.
    • Aloys: is Autodesk waiting on any open source projects for Python 3 support in Maya? Gordon: will have a beta version of Python 3 Maya soon, but official releases in this calendar year won’t have Python 3 yet. Aloys: may need preview / beta version for testing with Python 3 builds.
  • devtoolset-6 deprecated since CentOS-7.7 ([https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!topic/vfx-platform-discuss/-_CPw1fD3c](https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!topic/vfx-platform-discuss/-_CPw1fD3c))
    • Seems to indicate that VFX platform should consider upgrading compiler?
    • Now running CentOS 7.7 for containers, but with patch to get devtoolset from CentOS 7.6
    • Should be discussed in context of VFX Reference Platform. Daniel: Aloys might want to present the work he’s done on the VFX Reference Platform email list.

Action Items

Next Steps

  • Follow up meeting: 17 October 2019