Link Search Menu Expand Document

ASWF CI Working Group

Meeting: 15 May 2019

Attendees

  • Daniel Heckenberg (Animal Logic, TAC Chair)
  • JF Panisset (VES Technology Commitee)
  • John Mertic (Linux Foundation)
  • Bernard Lefebvre (OCIO / Autodesk)
  • Brian Cipriano (Google / OpenCue)
  • Jeff Bradley (Dreamworks)
  • Michael Dolan (OCIO / SPI)
  • Aloys Baillet (Animal Logic)
  • Larry Gritz (SPI)
  • Trevor Thomson (Blue Sky Studios)

Agenda & Notes

ASWF CI Goals for Year 1 [0:00-0:05]

  • 6 projects
  • CI with VFX Reference Platform dependencies
    • Commercial components
      • Build
      • Test
  • Stretch goals:
    • Downloadable and installable artefacts (with signing?)
    • Windows, Mac support
    • Possibly GPU support

Timeframes

  • CI Platform decision: May
  • Project CII badges: (Security and static analysis) June
  • Dependency management: July

CI Platform: Confirm Azure? May [0:05-0:25]

  • LF Rel Eng Update (Andrew)
  • OpenColorIO on Azure Pipelines (Michael)
    • Got Linux builds set up in Azure
    • Pretty positive experience
    • Azure DevOps interface presents an online IDE
    • Various plugins to streamline things like CMake
    • Can edit config file directly in Azure DevOps UI
    • Working in ci test branch
    • Working out kinks
    • Currently using Conda for dependency management since available on all platforms
    • Daniel: any obstacles identified? Michael: nothing in particular, a strange issue when specifying C++ 11 and some tests built with C++ 17, but could be a configuration issue.
    • Next goal to get Mac and Windows builds. Goal is to set cross platform build as much as possible, with test matrix that supports all platforms.
    • Daniel: does it look difficult to get the RHEL Developer Tool Set into the build environments? (for VFX Platform compliant builds). Michael: there is Docker support, but possibly only for Linux. Each task in Azure pipeline is its own process, can have persistent environment across tasks.
    • Aloys: managed to configure Ubuntu VM to run Container images, got a build that runs through a Docker Image with CentOS 7 base with DTS 6.1, build takes about 6 minutes, 22 minutes for the test. Was reasonably straightforward. Needs a somewhat compatible Docker image, but fewer restrictions than CircleCI docker images. Have to make sure you don’t write outside your Docker user. Used Conan, not Conda yet.
    • Aloys: Docker container support for Windows, but probably not needed, more relevant for Linux.
    • Building in Linux container should address issues run into with OpenVDB build on Azure.
  • Artifact storage / Nexus
  • Credentials e.g. SideFX clientID and secret key
    • Azure key vault
    • Should be able to manage artefact push to repository as well as access to any external services with costs associated
    • Also any licensed software?
  • Other criteria
    • Pricing for sufficient resources to run tests?
    • Machine configs may be quite different for tests? OpenVDB tests?
    • OCIO builds only needs GPU machine to run the test suite, not for the build:
      • Need a Packer recipe to build a GPU Azure VM
      • Could we get any support from Microsoft, financial support for GPU VMs?
      • Azure Pipeline agent is open source, would have to be added to the VM
    • Brian: what about multi-system OpenCue testing? Daniel: OpenCue is currently leveraging some of the existing LF / Jenkins infrastructure.
    • Should be possible to run unit tests / integration tests from other cloud providers
    • Aloys: Azure Pipelines has K8S service, so that could be leveraged.

Project CII badges: June [0:25-0:30]

  • Security and static analysis
    • Updates from last TAC meeting
    • Freedom to make practical / common sense decisions
    • Not too many details on the mechanics of how this happens
    • Slated for our next phase of work
  • Dependencies?
    • Difference on component being developed (ASWF project) vs components included in the project.
    • Emphasis on analysis of dependent components. Do we need to do vulnerability scan of our dependencies, or just our own projects?
    • Micheal: getting dependencies out of our repos is a way to rely on upstream
    • Daniel:We typically defer our specific dependency versions to external entity (dependency management system, VFX Reference Platform)
    • Michael: we could run static / dynamic analysis on the dependent components as well
    • We will probably need a “meta” project to build reference builds of the components we rely on (Boost…) so all of our projects can pull in those builds. That would be a place to run analysis on those components?

Project CI requirements [0:30-0:50]

  • OpenVDB
    • Update from Dan
    • Asking for some temporary funding to increase CircleCI memory limits until they can transition to Azure Pipelines.
    • Aloys: could help with transitioning to Azure / container based build.
    • Daniel: they are trying to work towards a release, so they may be reluctant to move to a new CI system from CircleCI
    • Daniel: could take a bit to get funding approved from the governing board
    • Not clear what the pricing would be to increase resources
  • OpenColorIO
    • Mark Boorer from ILM looking at embedded dependencies issues, so should have progress to report there soon.
  • OpenEXR
    • Larry: TSC established, but very minimal. Too easy to not make quorum, so trying to expand the group. Meeting pushed due to vacations. Initial items are mostly housekeeping issues, transferring mailing lists (in progress), moving GitHub repo, CII badge requirements.
  • OpenCue
    • Brian: TSC had first few meetings, dealing with project transfer issues. Highest priority is to unblock the repository transfer to ASWF GitHub organization.

Multiplatform Versions

  • Which versions of major components should we use on Windows, macOS? Same as Linux / Reference Platform?
  • Should get guidance from commercial DCC vendors since they have to support multi platform builds.

Action Items

  • Circle CI paid resources for OpenVDB (Daniel)
    • Alternatively assist with OpenVDB on Azure (Aloys)
  • GPU instance dispatch and costing on Azure (Andrew, Daniel)
  • Nexus Conda / Conan plugin investigation (JF)

Next Steps

  • Follow up meeting: 29 May 2019