Link Search Menu Expand Document

ASWF CI Working Group

Meeting: 16 September 2020

Attendees

  • Daniel Heckenberg (Animal Logic, TAC Chair)
  • Jean-Francois Panisset (VES Technology Committee)
  • Aloys Baillet (Animal Logic)
  • Andrew Grimberg (Linux Foundation Release Engineering)
  • Brian Cipriano (Google / OpenCue)
  • Christina Tempelaar-Lietz (OpenEXR)
  • Larry Gritz (Sony Imageworks / OpenShadingLanguage)
  • Marshall Elfstrand (Apple Developer Relations)
  • Michael Dolan (OpenColorIO)
  • Simran Spiller (OCIO documentation)
  • Sean Looper (AWS)

Agenda & Notes

  • First meeting in 2 months due to Open Source Days, skipping that meeting

ASWF CI Goals for Year 2

  • GPU Build & Test (success!)

    • Using AWS CodeBuilder, just in time for Open Source Days / SIGGRAPH

  • Mac, Windows & Linux (New focus)

    • Including support for VFX Platform 2021 (recently finalized)

  • Packaging / Distribution

    • Platform ecosystem

    • Ties in with cross platform discussion: what is the equivalent of Docker containers for other platforms

  • Testing with commercial components

    • Leverage GitHub organization secrets

Follow ups

  • GPU Build & Test

    • OCIO success!

      • Outstanding work: builds for PRs

    • Document / template example setup?

    • Michael: works well, run on every branch merge. Buid on that AWS machine is very fast due to powerful platform. Runs in under 10 minutes, including all the tests.

    • Good assurance that the GPU build doesn’t break, especially as OCIO enters stabilization process.

    • Setup was straightforward once the logistics were worked out.

    • No build failures since, very consistent.

    • Andrew: build on merge, so always “behind the ball” if broken code is merged. Does OCIO get alers? Michael: you can subscribe to notifications for certain failures, but for not just keeping an eye on it.

    • Working out the GitHub secrets to run on a PR would be beneficial (request with GitHub). It’s on their roadmap.

    • 3 weeks ago GitLab added that support (forked repo can run merge requests in context of parent repo, but requestor has to be a “developer or better”, which in our case would be someone with committer rights). At least you can do it in the context of your own fork. Hopefully GitHub will have this before the end of the year.

    • Michael: once that’s supported it will be easy to support

    • Daniel: need to document process for setting this up in template project. Point to OCIO implementation. Does require a service request to LF releng to set up the CodeBuild instance. LF releng points to the label of the (ASWF) build container to be used.

    • Also GitHub project configuration (requires correct access rights), just a GitHub Actions setup, and a secret attached (using an organization wide secret). Actions being used are also open source.

    • Almost motivation for projects to transition repos to ASWF: OTIO still under Pixar, OSL still under Imageworks (waiting for CLAs).

    • Larry: after breaking the OSL builds on the GPU path, started down the route of getting GPU stuff to build, wanted to use the ASWF containers without GPU, needed the NVIDIA Optix libraries, talking to Eric Enderton, looking at what it would be required to add Optix to our containers (less straightforward then the CUDA libraries): NVIDIA helping with this.

  • VFX Reference Platform CY2021

    • Mac: Min. OS version: 10.13

    • Win: Visual Studio 2017, Windows SDK v10

    • Larry: discussion about sunsetting older versions of projects called out by previous years of VFX Reference Platform. Could the projects agree on this? Driven by which version of the DCC apps are still in wide use. It would definitely be useful for project maintainers, as well as which container versions we need to keep supporting.

    • Position of Reference Platform is that it is currently “out of scope”, but could be tackled in the future.

    • Larry: would be good to know what’s explicitly “out of scope”? CMake versions? Getting all our projects to build in the same environment definitely helps.

    • Aloys: ASWF docker containers supports back to version 2018, since you can’t find older versions of Developer Toolset. Larry: Imageworks still has shows running older versions of Maya / gcc 4.8, that can require some complicated hacks to support.

    • Aloys: gets really hard to keep all the packages working, but individual packages may want to keep longer history (such as OpenEXR). Having a single container where everything works is tricky, at some point you can’t have CUDA working on older platform “year”. Larry: probably not a single answer, maybe someone has to maintain a set of 2017 libraries for Maya, but running in a modern environment.

    • Daniel: asking Aloys to articular some kind of proposal to the TAC, lay out the main constraints, requests such as adding clang-tidy (may require re-releasing all the images). Larry: do you need to do that? If a project requires static analysis, could be done only on the current platform.

  • ASWF-docker updates

    • Aloys: OCIO / OIIO circular dependency, GitHub issue / PR. Need to not break existing assumptions from OCIO, since OCIO needs bits of OIIO (so incomplete components of OIIO in OCIO container), but tricky to do without breaking the OCIO CI image. There was a PR with some commits to help, but not done yet.

    • Aloys: ideal case is to break the cycle.

    • Michael: it would be sufficient to not build the apps in the container and break the cycle. Aloys: downstream users may expect the full OCIO / OIIO to be built.

    • Larry: are our containers trying to do “too much”, are they trying to be packaged artifacts, or just a CI build environment? We know that people using CI on other projects will want an OCIO with OIIO included since they write tests that expect that to work. But the stand alone binaries may be special purpose. But if users are consuming the packages as “distribution packages” it’s an other story.

    • Aloys: there’s a PR for a set of “runtime” containers with only the binaries, good candidates for distributing complete OCIO / OIIO utilities. Could this be a “stretch goal” to distributing all the binaries for our projects? Can also be used to drive Jupyter notebooks. This is where you would want to full versions of these projects. That work is sitting in a branch for now. Was waiting for runtime GPU support to release this.

    • Ticket with NVIDIA about OpenGL / CUDA Docker images since there’s a CVE against CUDA 11. So for now VFX 2021 images don’t have CUDA since need CUDA 11 for the gcc version, should happen in next few weeks. When that becomes available will rebuilt the VFX 2021 images. Would also be a good time to integrate OCIO / OIIO / OSL with CUDA 11. Larry: have tested against CUDA 11.

    • Docker Hub repository changes. Andrew: attempting to get ASWF under the “Open Source” program. We have “aswf”, “aswf-testing” and “aswf-?”, we would have to purchase $900 / year. Aloys: we only need to worry about retention for “aswf”, not the others. Andrew: the announcement from Docker Hub affects retention policy and number of times an image can be pulled directly (limited to fairly small number, maybe 5 per hour? day?). Could seriously affect CI if we have any number of systems. So we want to at least sign up for the $300 / yr plan. Seems like the best route. Sean: have we looked at AWS ECR? AWS could offset the cost for that. Andrew: this would work great for our CodeBuild stuff, but by default Docker won’t talk to any container registry other than Docker Hub, there are workarounds, but it is not trivial issue. Docker has gone on record that they won’t make it easy for Docker to access other container registries. It is possible to put a proxy in front of another registry, but our end consumers who may not be part of ASWF can only get our containers easily through Docker Hub. Andrew: the cost is per named user, there are team accounts, Aloys is the only developer who has direct access. The secret in GitHub Actions counts as a “user”. We can stay at a relatively inexpensive setup as long as we don’t add a lot of users with registry access.

    • Daniel: cost is negligible, we should go with the paid “Open Source” mode. Aloys: they count from a user point of view, even for pulls, unless they detect CI use cases, but no details provided. There may be a FAQ somewhere that states they will try to be “CI friendly”. Also a “sub layer” of a container won’t count against the pull count. Sean: a discussion in context of USD: industry as a whole could derive execution environment from the ASWF containers, is that something we would lose out on with a Docker Hub paid plan? Would we no longer be able to provide that in the long run? Andrew: this is difficult to answer right now. LF releng is tracking this issue, other LF projects are also using containers, so this is an issue that affects everyone. Should we stand up our own registry to keep older containers around so they don’t die? What should be the retention policy? Should we keep every container version we’ve ever released?

Current activities

  • Windows CI

  • Mac CI

    • Daniel: pinged Marshall about questions of increase availability of Mac in CI environment, access to Apple Silicon based machines.

    • Larry: a couple of projects have gotten feedback from Apple to access Apple Silicon systems on an individual basis

    • Some intricacies of how Xcode presents itself differently on different hardware

    • Marshall: some investigation of what MacStadium / Orca is doing (MacOS as a VM inside a Docker container). Daniel: in practice we are just using the GitHub Actions provided Mac runners.

    • Daniel: some projects have contacted Apple directly, but would be good to have all ASWF projects be able to run CI builds against Apple Silicon. Larry: will become a requirement for every project, since DCCs using our projects will require native support for Apple Silicon.

    • Marshall: the Developer Transition Kits are the only hardware available, MacStadium does have some. Larry: Apple pointed to MacStadium, there is a program where Apple may offer credits against MacStadium for open source projects. Marshall: not part of the relationship with MacStadium. Larry: part of the “Creative Pro Relationship” (?) program at Apple.

  • Package management

    • Dependencies

    • Build deployment

Project Specific Goals / Problems

  • USD-WG goal (Daniel)

    • “Cross-platform build recipes / CI for USD”

    • Tests, GPU

Next Steps