ASWF CI Working Group

Meeting: 03 April 2019

Attendees

• Daniel Heckenberg (Animal Logic, TAC Chair)

• Jean-Francois Panisset (VES Technology Committee)

• Trevor Thomson (Blue Sky Studios)

• Andrew Grimberg (Linux Foundation)

• Michael Dolan (Sony Imageworks)

• Jeff Bradley (Dreamworks)

• Larry Gritz (Sony Imageworks)

• Aloys Baillet (Animal Logic)

• Patrick Hodoul (Autodesk)

• Dan Bailey (ILM)

• Gordon Bradley (Autodesk)

Agenda & Notes

• Welcome to Andrew Grimberg, Manager for LF Release Engineering group, taking over for Thanh.

ASWF CI Goals for Year 1

• 6 projects:

  • Environment configuration?

  • CMakeTools?

• CI with VFX Reference Platform dependencies

  • Commercial components

    • Build

    • Test

• Stretch goals:

  • Downloadable and installable artefacts (with signing?)

  • Windows, Mac support

  • Possibly GPU support

Circle CI vs Jenkins

• Decision rubric: discussions between Daniel and John Mertic on what the decision vectors should be:

1. Costs ( both hosting and support )

2. Capabilities ( OS support, GPUs )

3. User experience

   1. Encourage good / modern software development practices

4. Infrastructure reproducibility: costs, Internet access, seems to be less important than previously deemed, both opinion of Governing Board and TAC

• Update

  • Activated CircleCI free for ASWF organization

  • Cost estimates?

  • On premises: Terraform does actually allow this via suitable providers aka resource APIs (including VMWare vSphere, OpenStack)

  • Meetings have not happened yet with CircleCI for GPU cost estimates, Andrew still in touch with Thanh so has access to information on what he was working on.

  • There is the possibility of running multiple CI stacks in parallel, but more work needed on job design to make them CI agnostic.

  • Who owns the license of the current Windows build VM? Server 2016 license is owned by the cloud provider (VEXXhost), license cost included in VM running cost.

  • Daniel: given the flux of CI systems, it does seem like being as “CI agnostic” as possible is a desirable goal. Andrew: current stack is very Jenkins centric, but goal going forward is to decouple from Jenkins, moving Jenkins-centric parts into Ansible.

  • Daniel: we are also looking at package management for the software we build.

  • Daniel: how much value to the releng team is the work we are trying to do, given the different tool stack of ASWF projects (mostly C/C++/GPU) compared to other LF projects (mostly Java based). Andrew: there’s value to both trying to solve issues “holistically” as well as specifically. The releng team has visibility across all LF projects and can see commonality, so they can help with approaches that are viable for all projects. Sees ASWF as trailblazing in some areas and consuming what releng team has done for other areas.

  • Dan: perspective from OpenVDB. Currently working on transition to CMake, and how it integrates with CircleCI. Not all that difficult, went faster than expected. Haven’t really used CircleCI “in anger” yet, but seems a better alternative to Travis and faster (no preset 50m limit for builds). Also memory footprint is an issue, easy to bump into limits. Current plan is to merge CMake / Circle while keeping Travis for now, eventually switch completely to CircleCI. It’s also important to test the actual CMake configuration itself, for instance downloading OpenVDB source and installing dependencies, either from source or from package manager, would like to eliminate issues where the source doesn’t build on a specific platform because of dependencies. For instance OpenEXR namespacing vs not, which version you get when you apt-get. CI should capture all the ways in which developers may want to build the package while still documenting the “official” way to build the software.

  • Daniel: seems to lead to need for coherent handling of build, packaging and runtime environment? Aloys: consistent way to “find” packages in CMake would help a lot. Dan: agrees, looking at a versatile “find package” mechanism, handling of versioning namespace, clever but not “too clever”. We should come up with a standard set of what ASWF projects should support in that respect. Should not be done in isolation on a per project.

• Azure Pipelines / DevOps (vs CircleCI from MS)

  • Supports all 3 platforms: Windows, Linux, Mac

  • GPU support: must be possible as Azure GPU VMs are available (specs and costs)

  • JF: Trivial attempt at building IlmBase, Windows build images have GitBash installed so could probably use bash instead of PowerShell for Windows builds https://github.com/jfpanisset/openexr/blob/master/azure-pipelines.yml

  • Daniel: need to capture license tokens / licenses for commercial software, Azure has arrangement with Autodesk where you can get VMs that already have Autodesk licenses.

• Google Cloud?

  • Cloud Build Linux only

  • GCP with Jenkins?

    • No Mac VMs

• Are we ready to give up full open source solution / reproducible stack?

  • Governing Board indicated that they would be comfortable to forgo this in favour of more developer / project utility

• Useful piece of Python code to manage encrypted credentials in Jenkins config.xml: https://github.com/tarvitz/jenkins-utils

• Need to have some realistic cost estimates to make a decision, will need Governing Board decision once there are hard costs involved. Hopefully should be able to present a decision for next Governing Board meeting a month from now.

• Aloys: would greatly help if the Maya DevKit could be repackaged as a Conan or Res package. Gordon: red flag is “avoid anything that sidesteps the EULA”.

SonarCloud vs SonarQube

• Update

  • OpenEXR example on SonarCloud

  • Was under investigation by Thanh

  • Andrew: SonarCloud is free to open source projects, evaluating whether to push all projects currently using SonarQube to SonarCloud.

Project CI requirements

• All

  • CII badge static analysis

• OpenVDB

  • Follow up: Download Houdini for plugin build & test?

    • CLI download tool repo was transitioned to Andrew.

    • JF: could lead to a common tool for downloading commercial apps. Should this be an ASWF project? Could be part of “CI neutrality”.

    • Dan: would prefer if it was maintained by SideFX. Thinks it would be preferable if we interfaced with API directly.

• OpenColorIO

  • Michael: still waiting on reworking external dependencies before they can move forward with CI considerations.

VFX Reference Platform Dependencies and Package Management

• OpenEXR / VFX Platform CI challenge

• Aloys: OpenEXR (VFX Platform 2019) on Circle CI with docker images via conan

  • Agrees with Dan on CircleCI memory limit is an issue, have to restrict to 4 CPUs when building IlmBase. Things that use templates extensively can be an issue, would be helpful to have more memory.

  • Created graph of jobs that captures cross job dependencies and upstream dependencies. Posted an image on TAC mailing list.

  • Tried similar on Azure Pipelines.

  • Created packages for OpenColorIO, NumPy, Qt, Boost, OpenEXR. Long term goal to build up to USD and MayaUSD. Still some packages to build but pattern is easy to reproduce. Took ½ hour for OpenColorIO.

  • “Find Package” in CMake can get packages from Conan without having explicit dependency (can decouple package manager from CI). Makes sense to clearly separate layers.

  • Would be very helpful if all vendors used the same Qt build / build recipes, would avoid vendor-specific, Qt issues he ran into.

  • Jeff: started looking at examples in repo, should be very helpful.

  • Aloys: starting to look at Docker on Windows, build the same packages on Windows.

  • Conan has support for multiple Mac toolchains?

  • Does Autodesk has expertise to share for Mac / Windows builds? Foundry does use Conan for those builds. Gordon: can forward specific questions to the internal build teams. We should put together a set of questions.

Next Steps

• Follow up meeting: 17 April 2019

• Meeting time / Timezone changes?

---