Link Search Menu Expand Document

ASWF CI Working Group

Meeting: 24 June 2020

Attendees

  • Daniel Heckenberg (Animal Logic, TAC Chair)
  • Jean-Francois Panisset (VES Technology Committee)
  • Andrew Grimberg (Linux Foundation Release Engineering)
  • Larry Gritz (Sony Imageworks / OSL)
  • Michael Dolan (OpenColorIO)
  • Aloys Baillet (Animal Logic / ASWF Docker)
  • Jeff Bradley (Dreamworks)
  • Brian Cipriano (Google / OpenCue)
  • Sean Looper (AWS)
  • Mike Starr (AWS)

Agenda & Notes

ASWF CI Goals for Year 2

  • GPU Build & Test (Close to production on OCIO)
  • Mac, Windows & Linux (New focus)
  • Packaging / Distribution
  • Testing with commercial components

Follow ups

  • GPU Resources
    • OCIO request for AWS setup JIRA ticket (Andy)
    • Michael: OCIO started working on this yesterday, set up credentials for AWS CodeBuild, working on build specs for CodeBuild to replicate OCIO builds. Will then work with Andy to get project set up. Seems with the integration in GitHub Actions, it will be straightforward, no need for multi-stage jobs to spin up and tear down GPU server. Passes many GitHub variables to CodeBuild side, a bit more manual management of grabbing the right commits from the PR branch.
    • Andy: will also need to explicitly pass any secrets required, but no need of secrets for the AWS side.
    • Michael: hoping to have something running by the end of this week. Andy: we will then have a template for adding to other projects.
    • Daniel: USD WG was looking at what value could be added with ASWF CI, a mechanism they could use directly to run their test suite which is GPU dependent. Also discussed running CI on the code snippets / example code we might have in the WG repository.
    • JF: should this leverage ASWF docker container, or USD “top of trunk”? Daniel: depends on the use case, is it meant to demonstrate code running against a USD release, or meant to augment the USD test suite?
    • Aloys: submitted a PR to build USD against Azure Pipelines last year with ccache support. Has not been merged yet. Builds in 16 minutes. USD may consider moving to GitHub Actions.
  • Windows
    • Proof of concept of building Windows container with GitHub Actions and WIndows version supported by GHA (JF)
    • JF: no movement there, no support for Windows container builds on GitHub Actions at this time. Andrew: that is correct. Also GPU is supported with CodeBuild only on Linux, including containers. Would need explicit instance management for Windows GPU, not currently supported by AWS CodeBuild (no GPU available on Windows).
    • Daniel: would using AWS CodeBuild be another way for us to have more control over Windows CI? Andrew: CodeBuild on Windows currently doesn’t support GPU or containers.
    • USD, OSL make GPU on Windows more important, OCIO’s requirement was originally mostly satisfied by GPU on Linux.
  • Mac
    • MacStadium / Orka
    • Code Signing?
    • Apple Silicon
      • Larry: any announcements from GitHub about ARM macOS instances? OSL is very Intel oriented, will need significant effort to get good ARM support.
      • No announcements about CI / resources
      • Larry: Travis started offering non-x86 instances, some ARM varieties.
      • Graviton instances in AWS CodeBuild? Sean: there is documentation that seems to indicate there is. May need to have access Graviton instances added to ASWF / LF Releng account. This would be ARM builds on Linux, but would lay the groundwork for ARM builds on macOS.
      • Larry: gcc / clang / llvm does support ARM
      • macOS will support dual architecture binaries as well as Intel emulation via Rosetta2
      • Sean: AWS has several projects to support Graviton, so definite interest there.
      • Will macOS 11 still support OpenGL? What about Metal / Vulkan?
    • Discussion on Xcode version on VFX Platform mailing list
  • ASWF-docker updates
    • Git v2
      • Done by Brian
      • Not released every new Docker image yet, only the ci-opencue images, waiting on feedback to release the others
      • Ensures that you end up with a working git repo inside the container (otherwise v2 of the “checkout” GitHub Action only does a copy of the code tree)
      • New images are working, haven’t tested the functioning git repo part yet, but will do so tomorrow.
      • Git v2.18 still behind the current version 2.28, but that’s the version supported by RedHat Software Collections.
      • Andrew: IUS yum repository is what LF uses for latest and greatest version. Latest version adds gpg checking, which is required by some LF projects. Also latest versions now support v2 version of the protocol, which is significantly faster for large repos (GitHub supports that protocol version). V2 protocol becomes default in v2.25. This is not the SHA-256 support yet, which has not been merged yet.
      • Aloys: also GitHub Actions does a shallow clone by default, which creates warnings in SonarCloud. Have to do a “git fetch”, there’s an option in the GitHub Action to pull full git history (SonarCloud will do a git blame to identify when a new warning is identified). Andrew: a 10 level deep clone seems to mostly satisfy SonarCloud. Not sure if the GitHub Action checkout action lets you specify the depth of a git shallow clone. LF uses 10-25 levels deep for Jenkins workflows. Brian: yes, you can specify depth of clone at checkout time.
    • Much cleaner way to release GitHub images, when you create a GitHub release, a GitHub action pushes the release. So more traceable way to go from a Docker back to the build process. So every merge to master no longer triggers a full release.
    • Added a tool to create a release (80 Docker images)
    • Discussion for OCIO / OIIO and circular dependency between the test / utility programs (discussion started by a PR). Will create separate ocio-tools and oiio-tools packages to break the cycle.
    • Larry: historical fluke, have been talking about solving the problem for a long time. Aloys: important to solve for CI of both projects. Would like to support full OIIO that supports OCIO. Also downstream for OSL. Work has started on this.
    • Larry: sequence should be: build OCIO library only, build full OIIO complete, then complete OCIO
    • Larry: not the last time we will run into such circular dependencies
    • Michael: not a huge deal to not build the OCIO apps. Aloys: but mostly important for downstream projects.
    • Larry: OCIO tools that depend on OIIO are either demos or very specific tools of interest mostly to color scientists. Michael: current ACES config uses some of those tools, but also mainly used by color scientists..
    • VFXPlatform 2021
      • Newer compiler testing (GCC 9, Clang 10)

Project Specific Goals / Problems

  • USD-WG goal (Daniel)
    • “Cross-platform build recipes / CI for USD”
    • Tests, GPU
  • Artifact storage (OpenCue, Brian)
    • S3 bucket, currently in use for OpenVDB
    • Brian: run test on commits to master, create docker images, tarballs. When you cut a release, point to a specific commit, and pull all the artefacts from that build. This was easy to do in Azure Actions, no way to do that on GitHub Actions. Seem to be getting somewhat close using hacky GitHub Actions REST API, but seems unsatisfying, and would be cleaner to push artifacts to our own storage. Happy to use our own S3 bucket?
    • Andrew: current S3 bucket is set as a “release artifacts” bucket, specific policies about mixing artifact types between policies. So could allocate a separate S3 bucket, or reach out to JFrog for a free Artifactory Cloud set up for ASWF. Supports everything that Nexus did, at the time Nexus was set up, Artifactory Cloud did not exist.
    • Brian: not sure would classify these artifacts as “development” artifacts, they just don’t have a GitHub Release associated with them.
    • Andrew: on merge of code, produce artifacts that get stored in a “snapshot repository” (in a Maven / Java environment). Then a staging repo for release candidate, then a release repository for the official releases (3 phase strategy).
    • Brian: as long as it’s not too repository could be OK, but also very easy to just use a S3 bucket. Andrew: would require having versions on artifacts so you can retrieve them back. Io.aswf.project.artifact-version for example. Could use git sha-1 as the tag / version. In Java / Maven can use snapshots, and resolve latest version of snapshot (otherwise use “coordinates” to refer to specific versions).
    • Andrew: turnaround to deploy Artifactory Cloud is around 2 weeks, a new S3 bucket would be around 1 week. We would set a timeout policy on the S3 bucket, recommend 30 day for development artifacts. Brian: not producing artifacts from PRs, only commits to master.
    • Andrew: LF releng already supports Artifactory Cloud for a couple of projects. JF: also supports other repo formats (Conda / Conan). Also both S3 and Artifactory could be both useful. Andrew: there is cost to JFrog for standing up an Artifactory Cloud, so we don’t want to request it unless we will use it. S3 usage would be Pay As You Go. Stick to a S3 dev bucket for now.

Next Steps

  • Daniel to send out proposal to make CI WG official, and deploy a Slack channel.
  • Follow up meeting: 22 July 2020