Link Search Menu Expand Document

ASWF CI Working Group

Meeting: 28 April 2021

https://zoom.us/j/757849640?pwd=QzE1K2hrL2FHSFhKK3h5Z3BWTFJsZz09

Attendees

  • Daniel Heckenberg (Animal Logic)
  • Jean-Francois Panisset (VES Technology Committee)
  • Aloys Baillet (Animal Logic)
  • Andrew Grimberg (Linux Foundation RelEng)
  • Christina Tempelaar-Lietz (Epic Games)
  • Ryan Bottriell (ILM)
  • Michelle Halliwell (Disney Animation)
  • David Aguilar (Disney Animation)
  • Sean Looper (AWS)
  • Robin Rowe (Cinepaint)
  • Larry Gritz (Sony Imageworks)
  • Sergio Rojas (Arena World)

Agenda & Notes

Follow Ups

  • “Clear Agenda” after a couple of presentation-driven meetings

  • Slack channel has been reasonably active, we need to figure out what we do about the 10K message limit, so we can record some topics from Slack in the minutes.

  • Any news from JFrog about OpenSource account for ASWF? (from Aloys)

    • Andy: just updated ticket opened by Aloys, finally have JFrog platform up and running, configuring to use LF IDs as SSO. ASWF will be one of the first two projects to use it. Not just Artifactory, but also XRay and access to Artifactory CI (full JFrog Platform). Our trial was going to expire tomorrow, but that’s been extended until their open source program starts. We may need to rework some things, since currently there are no limitations as to what we can put in there, but there will be limitations when the open source program kicks in for real.

    • Daniel: XRay might be interesting? Andy: it’s a code analysis system, no LF projects currently use it, most familiar with NexusIQ (competitor to SonarType). Dependency Analysis and Static Analysis. Need to set up SAML logins.

    • Daniel: have looked at XRay for dependency analysis. Andrew: use NexusIQ for Java projects, since that’s where it’s strong. JFrog platform was built as competitor to SonarType. XRay will do Docker container analysis. Could also use NexusIQ. Reverse problem to package management: disassembly and analysis of artifacts / packages, chain of custody. Ryan: useful to look at XRay, but the CentOS 7 image already has vulnerabilities in it. XRay maintains its own list of CVEs and list of public CVEs, there may be issue they have specifically found. You will end up with a list of public and XRay specific CVEs that may or may not have yet been made public. Only core contributors have access to NexisIQ reports since their CVEs are more specific than the public ones. A NexisIQ report may point to a public CVE, but may provide additional information such as actual fix for a problem, more actionable info. Ryan: XRay is much more on the package side, you don’t really point it at your source code, but it will list all your dependencies. Daniel: we can discuss this more next time, Andrew, can you show us some of what it can do at the next meeting? Anyone else is free to dive in as well.

  • GitHub Actions Contributor / PR screening:

    • https://github.blog/2021-04-22-github-actions-update-helping-maintainers-combat-bad-actors/

    • Changes to try to reduce unauthorized resource consumption. Andrew: spoke to GitHub day before they released this. Ever since GitHub Actions was released, 30% usage of cryptomining, apparently this is consistent across free CI platforms. Mostly “drive by”, since you can specify GitHub Actions in a PR, so first step is that PRs from people who haven’t contributed to a repo yet have to be pre-authorized. Larry: what are the mechanics of how this works? Andrew: GitHub looks at PR and commits in a repo, and if the account raising the PR doesn’t already have commits in the repo, in the PR it will show “awaiting authorization”, and committers to the repo will get email notification, and a button that says “authorize and run”, any further changes to that PR will also block until they are re-authorized. So there’s potential for someone to first propose a valid PR, then a subsequent one that does something nefarious, but that’s unlikely. Currently no way to edit the list of committers, but GitHub is trying to get this out the door, as the percentage of cryptomining overhead is climbing.

    • Andrew: see this as a means to authorize certain users to run certain types of workloads, for instance GPU workloads as an extension.

    • Larry: would be interested to see how much work an attacker is willing to put in to “gain our confidence”.

    • Actions directory is not hidden, it is visible (it is a dot directory). Larry: you do want people to be able to contribute to the CI workflows.

  • New version of Cinepaint coming up, should I put a screen shot? Daniel: yes, of course, not CI specific, but in general #news channel.

ASWF CI Goals for Year 3

  • GPU Build & Test (success!)

  • Mac, Windows & Linux (New focus)

  • Packaging / Distribution

    • Aloys: no activity in last month on aswf-docker, no new issues. Will create a GH issue for Wayland support. Having access to Artifactory will trigger some experiments. Will ask Andrew to setup generic storage as well as Conan and Docker images, not clear why we would use Docker in Artifactory since we have access to free Docker Hub, but could be good to have it.

    • Looking at presentations on spk and others, we may want to start using packages to build these Docker images more flexibly. Have experience with Conan, could rebuild Conan images identical to Docker ones. Having Conan inside the Docker images would make it simple to add additional components inside these Docker images without having to pre-bake additional Docker images.

    • Will have to see what the limits are on the open source Artifactory. The Qt package by itself is a few 100MBs, Clang and LLVM are quite large (1GB).

    • Larry: Imageworks spk system is going to be open sourced, as of today have added first set of external partners to the repo. Trying to stage it so that the initial “out of the box” impression doesn’t leave a bad impression. Looking for people who will help take it to a MVP level outside Imageworks. In the coming weeks will have more news about what needs to be done and how it can be opened wider. Plan is to be fully public in not too long.

    • Aloys: this is timely and exciting, happy to look at it from point of view of ASWF. At this point it would be Linux only, trying to not limit the platforms we can maintain, but eventually spk should support other OS. Larry: stated goal of open sourcing is to get more hands / eyes on the problem to hopefully support Windows / macOS. Aloys: interested in looking at how to build these Docker images, and come up with recipes across the community. Is spk using Artifactory? Ryan: shared repo is NFS for now, but goal is to move to Artifactory. Aloys: seems like a high priority requirement, want to be able to try different containers “at the last minute” to be able to download pre-built components. Larry: that was the original goal. Ryan: this isn’t too far away, could come up with this kind of integration, built with idea of supporting multiple backends. But will need to figure out which ones need to be supported.

    • Daniel: one of the blockers for looking at Rez was the need to make CMake files much more general to support Rez, or make them specific to support Rez. vcpkg advice from cppcon was “don’t try to solve package management yourself, make your project easy to package”.

    • Ryan: coming up with a standard package description format would be very helpful. Could shared recipes in a central repo as well as binary repository.

  • Testing with commercial components

    • Tool to download Houdini releases now lives under Andrew’s user in GitHub, written in Go. Will accept PRs.

  • Secrets management

    • Andrew: the Magma project in LF has asked for secrets management, evaluating Hashicorp Vault which is now available as a cloud product, could be offered to projects. Could split the cost across multiple projects if that made sense.

    • Could be interesting for Infrastructure As Code components (OpenCue?). What are the CI needs of a project like OpenCue?

    • LF Releng uses https://www.passwordstore.org/ with GPG in conjunction with Terraform, which is completely free. But does require extra scripting. Scoping is pretty easy, have 3 separate scopes in password store.

  • Formalization of this WG

    • No changes since last meeting

New Items

  • Qt5 forks, Wayland

    • License changes affect use of Qt after version 5, caused a fair amount of discussion. How will vendors of commercial software adjust their usage of Qt? We use Qt both in commercial DCCs and in house tools. One outcome is a fork of the project, do we have a coherent sense? Autodesk has spent time consulting with stakeholders.

    • KDE will maintain a fork / branch: KDE long term Qt5 branch

    • Has anyone started using Wayland in our industry? It is where mainstream Linux is heading, there are some useful steps being taken for XWayland integration. Larry: at last week NVIDIA discussions, some GPU features don’t have an obvious mapping, such as utilizing multiple GPUs on a single machine. Not necessarily clear how we get from today to a Wayland future.

    • Specific question as to whether the Wayland package should be added to aswf-docker containers? Aloys: shouldn’t be all that difficult, looked at it while working on runtime Docker images (still working on those). Qt/Wayland might provide easier ways to run GUI apps in context of Docker. Should be reasonably quick to do.

    • MaterialX got accepted in incubation stage, a new project with GPU / cross platform requirements, will bring some CI challenges / opportunities.

Action Items

  • Daniel: WG formalization / split into own repo

  • Andrew: Summary of JFrog Xray for ASWF use cases

  • Daniel: Raise QT5, Wayland topics to TAC

Next Steps