Link Search Menu Expand Document

ASWF CI Working Group

Meeting: 21 July 2021

https://zoom.us/j/757849640?pwd=QzE1K2hrL2FHSFhKK3h5Z3BWTFJsZz09

Attendees

  • Jean-Francois Panisset (VES Technology Committee)
  • Aloys Baillet (Animal Logic)
  • Ryan Bottriell (ILM)
  • Larry Gritz (Sony Imageworks)
  • Jeff Bradley (Dreamworks)
  • Sergio Rojas (Arena World)

Apologies

  • Daniel Heckenberg (Animal Logic), WG Chair

New items

  • AWSF JFrog Artifactory status: seems like it’s up and running? How do we get to it?

    • Up and running, good news. Cannot push anything yet, only for GitHub, so only for releases from GitHub.

    • What repository types are enabled? Conan generic, Docker images.

    • Aloys has logged in, nothing in there yet, don’t have rights to push yet. Two Conan repos, one normal and one dev. Created a personal Conan repo on public Artifactory from JFROG to see how to try pushing and pulling, once there will be packages ready, they will be pushed from GitHub. There’s a GitHub Actions secret to be able to push to artifactory servers.

    • https://linuxfoundation.jfrog.io/artifactory/aswf-conan/

    • Not clear how much space we have, or indication of what the limits are.

    • Right now, aswf-docker images pull each package as a “sub image” (docker copy) which pull from Docker images that contain only one package, have to pre-build all these Docker images. So can pre-build every combination of images (clang-7, clang-8). Going forward, every package will be published as Conan package, can then create a small Conan file that specifies which package to pull, so can do more “a la carte” CI jobs.

    • In first step, want to be able to rebuild existing images with minimal changes that don’t impact outside users, once have ported over most packages should be able to keep going with normal CI Docker images, as if nothing changed, but will have access to Conan command. Will be able to just pull the base packages and the packages you want.

  • Conan and ASWF Docker containers

    • Right now, aswf-docker images pull each package as a “sub image” (docker copy) which pull from Docker images that contain only one package, have to pre-build all these Docker images. So can pre-build every combination of images (clang-7, clang-8). Going forward, every package will be published as Conan package, can then create a small Conan file that specifies which package to pull, so can do more “a la carte” CI jobs.

    • In first step, want to be able to rebuild existing images with minimal changes that don’t impact outside users, once have ported over most packages should be able to keep going with normal CI Docker images, as if nothing changed, but will have access to Conan command. Will be able to just pull the base packages and the packages you want.

    • Builds are still encapsulated within a container, but then results are exported to Conan.

    • Before, the build of the software was part of the “docker build” step, now the Docker images needed by Conan are minimal (just OS + conan package), “docker run” mounts the Conan cache and recipes, and build happens as “docker run”. Still running inside the container, but not as part of the build. Can inspect packages as part of the running OS.

    • Working on base packages: Python, Cmake, TBB, working on Boost, then will submit draft PR. Hoping to get some help, and hopefully fix up OCIO / OCIO circular dependency. Not sure yet if have to do everything up front before it can be used, or if it can be done incrementally. Will require a few months of work if have to port all packages up front.

    • Seems like a worthwhile effort. Picked Conan, since Artifactory supports it. Could use any other package management system that is cross platform. Foundry uses Conan and has recipes, seemed like least risky.

    • Will need a strategy for the transition, don’t want to break existing CI pipelines. Maybe we can rename images and have the new, more modular images called something else.

    • Some unanswered questions on Conan: has to be installed as a PIP package since it’s written in Python, but can’t have Conan dictate the Python version (since want to build Python), can use py-installer to install Conan into a different environment. So can build Python 3.7 from a Conan / Python 3.9 installation, that seems to work. Want to make sure to not pollute the result due to the details of Conan implementation.

    • Lots of small details to work out before it’s ready to go. Can either rebuild all existing images to be identical to the current system, or if that’s too hard / too risky, may end up changing name for new set of images.

    • Hoping for the identical images to make a transparent transition.

    • Looking at interaction with spk, Conan brings everything into a folder, but doesn’t tell you how to run it, it’s more a developer than a user tool, it seems like a good tool for CI pipelines, right balance, doesn’t force how to write a CI. Ryan: agree with the assessment, Conan doesn’t do distribution, but don’t need that for building. Aloys: also supports Windows / macOS, and we have “free” storage, nice combination of attributes that make it a good solution.

    • Will need to look at code signing

    • Currently ignoring pre-built Conan packages on Conan Central, like zlib, low level packages, reusing some of the recipes to avoid reinventing the list, but want to rebuild ASWF packages the same way we were building them previously. Putting them into a location with specific requirements, need VFX 2019 in your Conan settings, you don’t know what ABI pre-built Conan packages were built against. Hoping that the low level CentOS 7 provide all the low level dependencies.

  • ASWF Docker containers multi-platform support?

    • With those packages posted to Conan, should be able to start pulling packages for Windows (but Conan will be opt in).

    • macOS “not in container” builds could still result in publishing to Conan repos. Perhaps on Windows as well, may end up being Conan only builds, without Docker involved. Would download all pre-build packages from Conan into the VM provided by GitHub Actions. The Docker image that contains all the required tools can be used. Can provide scripts to install Conan, it is already part of the standard GitHub Actions builders, but may still want to install a specific version for the specific Python version we need, details to be resolved.

    • Docker image become optional at this point, don’t have to use them, they are necessary on Linux for the right ABI / support CentOS 7, but may be able to bypass that on Windows / macOS.

    • Hopefully able to look at the WIndows builds towards end of year. Any help is much appreciated. New employer uses Windows a lot, so will provide incentive.

  • Libraw and additional libraries

    • Should libraw be part of the base packages? Different versions included with different versions of CentOS 7. Right now it picks the one from the OS (latest 7.9), we can’t protect against the whole OS of course. Should “ASWF-adjacent” libraries such as libraw, libtiff be built explicitly. Approach for low level packages was “use whatever CentOS 7 comes with”. Some vendors build their own libpng, libz… Jeff: seems reasonable for the canonical ASWF CI, there’s so much variation even in well specified packages like Qt 5.12, can be different between Maya and Houdini for instance. Will never be able to control everything explicitly. Larry: rely on these packages for OIIO, test against a range of them, have scripts to test against other versions (latest, oldest), if the Docker containers have only one choice, need to rebuild anyway. Aloys: will have to pre-build those versions anyway. Conan lets you download the recipe and source and do the build locally, so maybe opportunities to add more flexibility in the system. Current solution is to “keep it simple”. Larry: it is an important case, we are trying to test that all of these packages will build “out of the box” on a generic CentOS 7 box, so we need to have that. Jeff: would be great to highlight those differences, sometimes miss warnings from CMake until something breaks. May want to elevate some of the more important dependencies.

    • Aloys: Whoever makes those version decisions makes them for many other people.

    • Could contribute a list of packages that could be controlled: libz, libpng, libxml2, expat, libtiff, libraw. Could provision Conan packages, but could create a another list of dependencies in the dependency tree (system versions vs controlled versions). So OIIO with system libtiff vs controlled libtiff ends up with 2 different binaries. Conan makes it possible to create parallel recipes, but it becomes a lot of management overhead. But we need it for Windows which has none of these libraries as par of the system.

    • Jeff: Don’t always have 100% upgrading to new CentOS version, getting everything synchronized can be difficult (some apps may require some machines to stay at older versions). Trying to sync libraw with CentOS upgrades makes more sense to control it yourself.

    • Aloys: on CentOS 6, res-ified most of the libraries since the system ones were way too old, had to be able to choose these versions. Then once on CentOS 7, could happily just use the system versions, but now it’s getting old again. Now using singularity images of the oldest OS we want to support, comes with a pre-defined list of packages, similar approach to Docker.

  • Still early to review anything, but will have draft PR to look at.

  • VS Code, Krita no longer work on CentOS 7, have to provide libstdc++ (can find those in ASWF containers).

  • Anyone tried the 2022 images?

    • Larry: got held up with Python 3.9, nothing wrong with the images, but wasn’t getting clean builds. Need more time to figure out the issues.

    • Aloys: let me know if you find any issues with the Python version.

Follow Ups