Link Search Menu Expand Document

ASWF CI Working Group

Meeting: 4 March 2020

Attendees

  • Daniel Heckenberg (Animal Logic, TAC Chair)
  • Jean-Francois Panisset (VES Technology Committee)
  • Aloys Baillet (Animal Logic)
  • Jeff Bradley (Dreamworks)
  • Michael Dolan (OpenColorIO)
  • Dan Bailey (ILM / OpenVDB)
  • Christina Tempelaar-Lietz (OpenEXR)
  • Larry Gritz (SPI)
  • Charles Fleche (Rodeo FX)

Agenda & Notes

ASWF CI Goals for Year 2

  • USD

    • PR from Aloys against USD to support GitHub Actions CI

    • ASWF USD Working Group meeting right after this WG

    • Aloys: already a branch to support Azure Pipelines, trying to move USD to Azure Pipelines. Not clear if they considered GitHub Actions. Probably makes sense to go to Azure Pipelines first because of UID issue.

    • Usd-build Python script will build and install all upstream dependencies, won’t run the tests.

    • Aloys PR disables the GPU tests since we aren’t there yet.

    • USD team seems to be keen on accepting this kind of proposal.

    • Daniel: CI expertise was identified as area where ASWF can help USD effort.

  • Python3 / VFXPlatform 2020

    • PySide 5.12 issues? See vfx-platform-discuss

      • Todo: Add patch or PySide 5.13 to docker images to test impact

    • Compiler / OS choice for 2021

      • Aloys: would be nice to move to newer compiler / platform / OS, but will cause issues.

      • Daniel: we contribute by identifying specific positive points to agitate for that transition.

      • Larry: projects should update their CIs to have a test case for newer environment, to help inform the conversation about any issues that might pop up.

      • aswf-docker: “top of tree” builds could include latest environment, but still struggling with Python 3 integration. CY2021 needs to be started based on current proposals. Being able to prototype impact of proposed changes would be very useful.

    • Python 3

      • Daniel: at last TAC meeting we agreed that we should have a WG around Python 3 transition, not just for ASWF projects but also internal projects

      • Currently no leader for this project, is anyone interested in leading this effort? Please reach out to Daniel directly.

  • Windows, Mac

  • Runtime tests with commercial components

  • Packaging / signing

  • GitHub / Azure Permissions

  • Exploring GitHub Action

    • Aloys: different user / permission model between Azure Pipelines and GitHub

    • Aloys: Dan / OpenVDB already running on GitHub Actions

    • Aloys: runs as root when running inside a container, whereas Azure configures a user on the fly, runs the steps / jobs as that user. Has impact on testing scripts, USD tests don’t all work as root (test for /etc writable). Should be possible to work around this. Maybe will need a different set of Docker images? Hopefully not.

    • Aloys: –user option can’t be used to change the default user inside a running Docker container, only when creating container. Seems like only one other person on the Internet / in Google was reporting a similar issue.

    • Daniel: have you (Aloys) found a contact person at GitHub? Aloys: haven’t tried to reach out yet, problem just popped up yesterday.

    • JF: https://github.com/jfpanisset/cloud_gpu_build_agent

    • JF: similar yet slightly different deployment for self hosted agent (even the version numbers are mostly lined up). GitHub Actions has an API for self hosted runners in particular to deal with generating a specific token to add a runner

    • GitHub Actions predefines environment variables GITHUB_ACTIONS (“true” when running in GitHub Actions), GITHUB_TOKEN (token for interacting with GitHub API) and GITHUB_REPOSITORY (current repo path), so a bit less setup needed in the CI environment.

    • JF: in GitHub actions if one matrixed job fails the others will be automatically terminated if there’s a follow up dependent job

    • JF: only one “level” of jobs in GitHub Actions, vs stages / jobs in Azure Pipelines. Syntax is very similar yet slightly different: sample Azure Pipelines YML vs GitHub Actions Workflow

    • Python is a “first class citizen” for running scripts, could be good for portability Windows / Linux

    • Daniel: has there been public discussions of future of GitHub Actions vs Azure Pipelines? Maybe a question for Andrew in lf-releng, they’ve had discussions with Microsoft on this topic.

    • Dan: a few small roadblocks looking at GitHub Actions. Trying to move Windows build from Appveyor. Issue of being able to download version of Houdini, what can you do about forked PRs, doesn’t pass secrets from master to fork. Trying to figure out a way to have secrets for forked PRs. Looking at caching in GitHub Actions to put Houdini download key in that cache. Have switched default to devel branch. Perhaps a manual approval step for running these PR checks?

    • Daniel: specifics of secret handling between master and fork, is that different between Azure Pipelines and GitHub Actions? Yes, Azure Pipelines has an option for “share secrets with forks”, even though it recommends not to do it.

    • Dan: have a working PR for moving Windows build from Appveyor to GitHub Actions, need to get lf-releng to switch off Appveyor integration. Michael: still using Appveyor for supporting older releases. Dan: Appveyor is running under Andy’s (lf-releng) account. Dan: hard to debug these issues when you can’t see the settings since you don’t have admin access.

    • Michael: no work yet on GitHub Actions for OCIO, but on the todo list. Patrick Hodoul noticed that some older OS versions getting deprecated on Azure Pipelines this month, so may need to be taken care of.

    • Larry: OSL has used GitHub Actions for a while, don’t see a reason to change, but haven’t had to access secrets yet. Has been pretty happy with it.

GPU Resources

  • JF’s PoC TAC list thread
    • Support for Azure, AWS and GCP
    • Working on GitHub Actions support

Project Specific Goals

Next Steps

  • Follow up meeting: 1 April 2020