Link Search Menu Expand Document

ASWF CI Working Group

Meeting: 1 May 2019

Attendees

  • Daniel Heckenberg (Animal Logic, TAC Chair)
  • Jean-Francois Panisset (VES Technology Committee)
  • Andrew Grimberg (Linux Foundation Release Engineering)
  • Jeff Bradley (Dreamworks)
  • Larry Gritz (Sony Pictures Imageworks)
  • John Mertic (Linux Foundation)
  • Aloys Baillet (Animal Logic)
  • Michael Dolan (OCIO TSC / Sony Pictures Imageworks)
  • Dan Bailey (ILM)

Agenda & Notes

ASWF CI Goals for Year 1

  • 6 projects
  • CI with VFX Reference Platform dependencies
    • Commercial components
      • Build
      • Test
  • Stretch goals:
    • Downloadable and installable artefacts (with signing?)
    • Windows, Mac support
    • Possibly GPU support

Timeframes? (Daniel)

  • CI Platform decision: May
    • What’s the cost of “picking the wrong solution”
    • We have pretty much converged on a SaaS solution
    • Different SaaS systems are more similar than they are different
    • Dan: would want to see one of our projects actually using one of the systems. “Incubation phase” for a CI system.
    • Daniel: As part of our goal for a decision by end of May, we will have one project using a SaaS system
    • Aloys: should this timeline be in our Jira. Daniel: good point, we haven’t really used the ASWF Jira (https://jira.aswf.io). Currently in used by OpenVDB project.
    • Andrew: credentials from identity.linuxfoundation.org https://identity.linuxfoundation.org/
    • Dan: can link Jira tickets to GitHub
    • OpenVDB is using Jira as a bit of a “mailing list” to discuss issues, but have to manually add participants to discussion. Andrew: requires a custom notification scheme. But out groups.io mailing list doesn’t support this.
  • Project CII badges: (Security and static analysis) June
    • Would like to graduate some projects to graduation stage before SIGGRAPH (one year anniversary)
    • OCIO in progress on static analysis, could be the first one
    • Requirements for project website
  • Dependency management: July

CI Platform Comparisons

  • https://hackernoon.com/continuous-integration-circleci-vs-travis-ci-vs-jenkins-41a1c2bd95f5

  • (Andrew) Update
    • CircleCI pricing for local instance / paid plan pricing is quite high
    • LF RelEng is moving towards Azure Pipelines, with optionally GitLab CI for specialized requirements
    • Experience in LF releng team is still mostly Jenkins, so any new system is going to be a new thing
    • GitLab CI may have some better API compatibility with existing infrastructure
  • Azure Pipelines is likely choice.

SonarCloud

  • (Andrew) Update
    • No update right now
    • Can have some interesting issues working against branches

Project CI requirements

  • All
    • CII badge static analysis
  • OpenVDB
    • New release coming up shortly
    • CMake update branch when in today
    • Migrated from Travis to CircleCI, plus AppVeyor for Windows
    • CircleCI seems quite a bit faster than Azure, but fewer builders (OpenVDB has many builds). Amount of memory is key criteria (OpenVDB builds are memory hungry).
    • Has been working on an Azure build, ran into some issues, not enough info yet to have a good idea on how well it would work for OpenVDB.
    • Haven’t started on CII yet, will do once upcoming major release is out.
  • OpenColorIO
    • Hooked up SonarCloud to OpenColorIO (in isolated branch / repo)
    • Created a dedicated branch to be able to iterate quickly.
    • Not getting any valid results, maybe because not using master branch? Looks like it only analyzes “new changes”.
    • Fairly straightforward to set up with Travis.
    • Look at ci_test branch in OpenColorIO GitHub (stripped down Travis setup for SonarCloud)
    • CI build is pretty simple, so could be good candidate to test out different CI system. Also they need a GPU instance.
    • Andrew: LF may be able to get more access for Open Source projects from Azure (in progress), including perhaps GPU
    • Mike: using a single CI system is helpful, although you might get better parallelism by using multiple CI systems…

  • OpenEXR

  • OpenCue

    • Brian couldn’t join meeting today, but no further progress to report

VFX Reference Platform Dependencies and Package Management

  • VFX Platform 2020 draft
    • Sent by Nick Cannon
    • Main focus is Python 3.7.x, deliberately not much else
  • OpenEXR / VFX Platform CI challenge
    • (JF) OpenEXR on Azure Pipelines with SonarCloud
    • We will need to come up with a set of SonarCloud warnings that need to be fixed vs ones that can be safely ignored / squelched
    • Andrew: you can set policy frameworks across an organization, so we can set those up ASWF-wide
    • Which platform to run the Sonar Cloud analysis against? Andrew: believes there should be a way. LF doesn’t run the for-pay SonarQube instances, so we don’t have support. Andrew to follow up with Sonar
    • Used conda to pull dependencies
  • (JF) Nexus3 Conan and Conda Plugins
    • JF to make a request LF RelEng Helpdesk to make it into tracking system.
    • LF releng watches disk space on various servers, currently 1- 2 TB on Nexus3 servers, will grow the storage if a VM starts throwing alerts. Have policies for purging unused artifacts.
    • Will require authentication for pushing, not for pulling unless we host proprietary artifacts (should not be in our case). Aloys: the Jenkins script to push to Nexus is quite involved. Would need to store tokens in Azure pipelines. Andrew: use Maven settings files to store credentials, and use the credential system of Jenkins.
    • Should we consider Artifactory Cloud? They support Conan directly. Rez is adding support for Artifactory. They have some free hosting for open source projects.
    • Aloys: we will need ASWF official accounts for all these services. Andrew: no clear solution for managing credentials across all these SaaS systems. Will need some kind of “sandbox” once projects get hooked up to the “official” CI system, artefactory

Next Steps

  • Follow up meeting: 15 May 2019