Link Search Menu Expand Document

ASWF CI Working Group

Meeting: 23 June 2021

https://zoom.us/j/757849640?pwd=QzE1K2hrL2FHSFhKK3h5Z3BWTFJsZz09

Attendees

  • Daniel Heckenberg (Animal Logic), WG Chair
  • Jean-Francois Panisset (VES Technology Committee)
  • Aloys Baillet (Animal Logic)
  • Andrew Grimberg (Linux Foundation RelEng)
  • Ryan Bottriell (ILM)
  • Larry Gritz (Sony Imageworks)
  • Mark Boorer (ILM)
  • Jeff Bradley (Dreamworks)
  • Sergio Rojas (Arena World)
  • Christina Tempelaar-Lietz (Epic Games)

New items

  • Presentation at ASWF Open Source Days? Deadline is today for session length and speaker confirmation (sorry!)

    • 15 minute talk?

    • Forwarded emails to Daniel

  • VFX 2022 docker images

    • Released soon

      • Started, but builds haven’t been launched yet

      • Issue with Sonar Client build wrapper was too old to keep working with the analyser, re-released all the pre-2022 Docker images with builds of Sonar. Christina confirms Sonar is working, no more messages about deprecated client.

      • Will now start the 2022 builds

      • Will no longer build update 2018 images and remove 2018 support from main branch, can always build from a branch if needed. Existing containers not removed from Docker Hub and won’t age out since we have a paid plan.

      • Some things haven’t quite been released yet for 2022, but are supposed to be released around SIGGRAPH timeframe

    • Requirements for materlalX project?

      • Additional dependencies? Reaching out to MaterialX (Daniel to reach out to Jonathan), maybe they can submit their own PR to add support?
    • No real pushback on decision to stop updates on 2018 versions, but not yet “officially” announced?

      • TODO Daniel: socialize no more 2018 image update

        • Docker Hub commercial means older versions will be retained and available
    • Updates on OCIO-related circular dependencies?

      • No updates
    • Any other “ASWF-adjacent” projects should have their own build containers / be included in ci-vfxall?

      • Alembic and PartIO are already in there.

      • What are we trying to get out of these containers:

        • Sanity check

        • Convenient sandbox for building VFX apps

        • Container to run the Jupyter notebook (outstanding PR with runtime set of images, would be much slimmer since it wouldn’t require all the build tools). A set of runtime images may make more sense to have them be complete. Vfxall could be a weird place to build?

        • Do we have specific examples to test against this? What are specific examples? MaterialX (shouldn’t be too hard, they already have a GitHub Actions to build).

        • Does vfxall have a succinct purpose defined? Yes, in the README

  • The Pipeline Conference: Jesse Lehrman is doing a DevOps talk and asking for participation from CI WG, but may have missed deadline?

    • Daniel: will talk to Jesse, Ryan is already particpating, will see if it makes sense
  • Rocky Linux 8.4 is out, anyone play with it yet? Apparently includes migrate2rocky Python script to help with conversion from other distros

    • Aloys: it’s an upgrade script from RHEL / CentOS 8, not 7

    • Andrew: building directly from RHEL sources, “bug for bug compatible”, does not yet do secure boot. LF systems have been using CentOS Stream 8 so far

    • Daniel: how much does the OS matter if we are using containers / filesystems like SpK? Vendors still need to release their apps against a supported OS. Maybe has to be a vendor led discussion. CentOS 7 will EOL in a few years. Aloys: VS Code no longer works on CentOS 7. Andrew: dropped Python 2.7 in LF Jenkins. Also new hardware support may require past CentOS 7.

    • Jeff: what are we using as container base image, is it still NVIDIA/CentOS? Aloys: containers rely on a few components from NVIDIA, but not that much, they could just be installed as RPMs, don’t need to rely on NVIDIA base image. Also depends on what the VFX Platform is planning to move to. The containers can help to explore this. Not clear where would find devtoolset for Rocky Linux? Andrew: Rocky Linux announcement says they have opened up their CI environment, everything they have done is forkable on your own. JF to look at devtoolset in context of RHEL 8. Daniel: devtoolset adds newer compiler versions, and mechanisms for deploying binaries built with newer compilers on systems with the older OS. Mark: devtoolset updates libstdc++? Daniel: no, maintains a static version with the newer symbols. Ryan: can build with gcc 9 on CentOS 7 and distribute binary on system that doesn’t have gcc 9 (for instance). Do you want to end up redistributing the standard C library?

  • CI requirements for Rust effort, can we help / coordinate?

    • Should someone from Rust WG be influencing what we do?

    • Ryan: was following Rust WG more closely at beginning. Cargo “wraps everything”, we could provide documentation on how to wrap with the Rust build system. There’s not necessarily something we could provide explicitly, apart from project maintainers making themselves available.

    • Daniel: should we be making efforts so that our CI builds can be easily picked up by Cargo? Also, provide standard template for GitHub Actions for Cargo (documentation).

    • Ryan: strong community around Cargo means these questions are mostly answered, as opposed to infinite variability in C++ world. Daniel: this relates to package management, and there’s a clear solution for Rust. Daniel: will follow up with Rust team.

    • Ryan: most Rust wrappers are re-distributed as part of the Cargo packages. The Rust WG will work directly with the projects, so unclear if there’s a ton of work for the CI WG to do.

  • Should CI WG have its own Confluence space? Do we have things to put in there? Or would built-in Wiki for “our own repo” be sufficient?

    • document recipes leveraging aswf-docker containers to compile “challenging” projects such as ffmpeg?

    • Part of the “WG formalization” project

    • Andrew: LF Confluence spaces aren’t blocked by Google, but we also don’t do anything specific to make sure it’s indexed. Information inside the Wiki should be discoverable, and Confluence is good about that. Not sure how good the GitHub Wiki is about that. LF Releng is not using a Wiki at all, using Readthedocs which is definitely searchable by Google, can validate URL links for instance when documentation changes are merged.

    • Ryan: would vote for repository based, clear place to update. Andrew: need a LF ID to login to Confluence and edit (can also link a GitHub identity).

    • Mark: not clear what kind of barrier to entry, a lot of users are put off by having to submit a PR just to submit documentation. Andrew: you need someone to verify / validate a Wiki on a regular basis.

    • Daniel: so there’s a question here, we should review what we have. We have documentation and artifacts in repos in our repos, our own repo will create a more direct entry point. Is there other information that should be presented in another form? Let’s make a decision as to what we want to use as a system so we don’t waste time on a documentation platform transition. Given the audience here, using PR mechanism shouldn’t be a problem, and don’t expect a lot of contributions from the outside. So Markdown files in a repo should work fine.

Follow Ups

  • Any news from JFrog about OpenSource account for ASWF?

    • Andrew: should have more time in a few days to work on this, next thing on list of priorities, should be done by next week.
  • WG formalization / own repo

  • Rocky Linux 8.4 and Development Tools

    • base gcc version in RHEL 8.4 / CentOS 8.4 / Rocky 8.4 is gcc 8.4.1: CentOS 8.4 gcc 8.4.1 source RPM
    • RHEL 8: Using GCC Toolset Red Hat Enterprise Linux 8 introduces GCC Toolset, an Application Stream containing more up-to-date versions of development and performance analysis tools. GCC Toolset is similar to Red Hat Developer Toolset for RHEL 7. GCC Toolset is available as an Application Stream in the form of a software collection in the AppStream repository….. Applications and libraries provided by GCC Toolset do not replace the Red Hat Enterprise Linux system versions, do not override them, and do not automatically become default or preferred choices. Using a framework called software collections, an additional set of developer tools is installed into the /opt/ directory and is explicitly enabled by the user on demand using the scl utility.
    • gcc toolset 9 includes gcc 9.2.1 : RHEL 8: GCC Toolset 9
    • gcc toolset 10 includes gcc 10.2.1 : RHEL 10: GCC Toolset 10
    • gcc-toolset-9 and gcc-toolset-10 are bundled with the CentOS 8.4 ISO as an AppStream:
    • gcc-toolset-9 and gcc-toolset-10 RPMs are included in the Rocky Linux 8.4 installation ISO
    • Install gcc-toolset with: yum install gcc-toolset-N
    • Run a tool in context of gcc-toolset: scl enable gcc-toolset-N tool
    • Run a shell with gcc-toolset as the default dev tools: scl enable gcc-toolset-N bash
    • In summary: gcc-toolset appears to be functionally very similar to the devtoolset mechanism in RHEL 7, but is more convenient to install since it is bundled with RHEL 8 and derivatives, including Rocky Linux 8.4

Action Items

Next Steps