Link Search Menu Expand Document

ASWF CI Working Group

Meeting: 11 November 2020

https://zoom.us/j/757849640?pwd=QzE1K2hrL2FHSFhKK3h5Z3BWTFJsZz09

Attendees

  • Daniel Heckenberg (Animal Logic, TAC Chair)
  • Jean-Francois Panisset (VES Technology Committee)
  • Aloys Baillet (Animal Logic / ASWF Docker)
  • Andrew Grimberg (LF Release Engineering)
  • Michael Dolan (OCIO)
  • Brian Cipriano (OpenCue)
  • Sean Looper (AWS)
  • Marshall Elfstrand (Apple)
  • Larry Gritz (OSL)
  • Simran Spiller
  • Jeff Bradley (Dreamworks)
  • Luke Titley (Dwarf Labs)
  • Christina Tempelaar-Lietz (OpenEXR)

Agenda & Notes

ASWF CI Goals for Year 2

  • GPU Build & Test (success!)

  • Mac, Windows & Linux (New focus)

  • Packaging / Distribution

  • Testing with commercial components

Follow ups

  • GPU Build & Test

    • Nvidia EULA for Optix SDK headers? (Daniel)

      • Discussion via email

      • Larry: NVIDIA provided download link, but no clarity on license? New Optix doesn’t have separate libraries, relies on CUDA libraries, just missing the headers. Really a question for NVIDIA.

    • Document OCIO setup in Template project (JF) -> not done yet

    • Any updates on GitHub secret handling? Andrew: no updates on that front.

  • ASWF-docker updates

    • VFX2021 / CUDA11

      • Aloys: lots of changes got released yesterday, seem to be breaking some workflows, Larry ran into issue with lack of namespace for environment variables to determine which version of components are exposed. Re-releasing containers with ASWF_ prefix for all environment variables, should fix the issue.

      • Updated to CUDA11, NVIDIA released updated base images, so VFX 2021 images are based on CUDA 11.1. Added proper license assignments in each package, updated Blosc which was quite far behind, but that broken OpenVDB, waiting for feedback as to which version they want.

      • Every Docker image has a README, and every image has full documentation in Docker Hub. Also make it clear that “ci-openvdb” doesn’t include OpenVDB, but is meant to built OpenVDB. Internally all images are based on Jinja2 templating. Also made available aswfdocker via PyPi, should be possible to generate your own ASWF images on demand. Tried to download PyPi package on Linux, seems to work fine, haven’t tried on other platforms. Wanted to reserve the ASWF name.

      • Daniel: good to make things more explicit based on questions / feedback. Acknowledging the grey area that Docker images reside in as binary distributions.

      • Aloys: some Docker images are “empty containers” that only contain the packages. Qt includes a long list of licenses, smaller packages are simpler. Header files may also include license copies.

      • aswfdocker is starting to look like a (very simple) package manager, considering how complex it should get before that’s “too far”. There’s a versions.yml that contains all the versions of packages included in a container, may add a proper dependency tree between packages, right now everything is just a flat list. Would be useful to capture that knowledge. Should be possible to keep the scope limited, we only need to manage the packages we are interested in.

      • Daniel: lots of thought and design behind Docker images, is the design clear enough from what’s in the repo, or should we provide documentation in the repo or in our Wiki? Aloys: would need more documentation for the aswfdocker tool, added a section in the CONTRIBUTING file as to how to release containers. It’s hard to know what people actually need. Daniel: can be easier to contribute to documentation separate from the repo?

    • Docker Hub repository changes.

      • Application for Open-Source status? (Andrew)

Andrew: only update relies to containers: haven’t gotten on Open Source Docker Hub program yes, other project has done it, but the Open Source program doesn’t look that promising. Anyone pulling from your open source namespace doesn’t count against your quota, but doesn’t help if you are pulling from other images. But still worth getting ASWF onto that program. Also initiated process of setting up a paid for account, once we are upgraded, our account should have any locations, same for Aloys account (part of that team).

No clear correlation with some performance issues experienced during CI builds.

Charges will come as a standard pass-through charge to ASWF.

  • Mac CI

    • MacStadium / Developer Transition Kits

      • JF to prototype integration between an ASWF project and Orka build (there is a GitHub repo demonstrating this integration) Ideas around the concept of a media creation company in the WFH era

      • Looking at using Vagrant + Virtual box to provide similar experience to Linux “build in Docker container” model

      • Andrew: should we use Packer instead? LF has library of Ansible playbooks that target using Packer

      • Daniel: building VMs for various versions of macOS may cause some licensing restrictions. Orka sidesteps licensing restrictions.

      • Marshall: check out the MacStadium open source project page JF: I’ve (just now) filled out the form, waiting for feedback. Daniel: requirements from MacStadium may rule us out.

      • Andrew: LF releng has managed MacMinis at MacStadium before, can be done, but team doesn’t have a ton of macOS expertise. Can be set up as a pass through cost to ASWF, allow the community to do the management. Whatever works best for the community.

      • Daniel: initial LF CI system was quite involved, hard to know where to start, may make more sense to start with the community, then hand over to LF releng.

  • Package management

    • Dependencies

    • Build deployment / runtime environment

  • Project feedback

    • Michael: everything is good with OCIO, GitHub Actions revised their policies so that Actions on forks will disable themselves after a certain duration of inactivity. Larry: can be turned back on, an “opt in” model. OCIO: some contributors would get emails that their Actions failed because of this, could lead to some confusion.

New Items

  • Created wiki space: https://wiki.aswf.io/display/CIWG

    • Good place to capture useful information, especially from Slack

    • May recreate to match other projects

    • Andrew: some other communities are using Confluence Wiki for meeting minutes, then lock the page once the meeting minutes are finalized. Confluence has a meeting minutes template. Daniel: lots of Confluence expertise at other studios.

  • We haven’t done much for Windows yet

    • Docker? Similar VM-based approach to Mac?

  • How do our projects interactive with distributions, package managers

    • How could we leverage existing package managers to get fine grained version control

    • Michael: aren’t we supposed to get a S3 bucket for arbitrary storage needs? Daniel: yes, we have one. Michael: OCIO would have a use case for artifact storage, would need this for some of their documentation workflows.

    • Brian: using that bucket in their CI to pass artifacts between CI stages. Andrew: it’s a private bucket, so won’t be able to access artifacts without the credentials (OCIO wouldn’t be able to access from ReadTheDocs for instance). No URL access, need to access with credentials.

    • Andrew: ReadTheDocs has an option to access pre-rendered material (RST format). Michael: currently testing that as a solution. Andrew: v3 API for ReadTheDocs allows building sub sites as needed.

  • ASWF Survey: Michael, there was a question as to which package managers should be supported by projects? Would be useful to know. Daniel: should be making progress on the survey soon, should be able to share an update later today.

Action Items

  • Do we have enough bandwidth / interest to tackle Windows? JF: will try to keep VM-based investigations for macOS to be as platform independent as possible. Andrew: there were some configurations to build on OpenStack on VEXXhost, could stand up a server on VEXXhost (need Semi Annual Channel version of Windows Server). Does AWS CodeBuild have Windows builders? Yes, so that could be an option.

Next Steps