The News: After a rocky eight months since the December 12, 2020 announcement that CentOS Linux builds would end in 2021, the community is starting to come out the other side, and the recent board meeting is a solid step on this journey. You can read the minutes of the board meeting here.

What is Really Going on with CentOS Stream?

Analyst Take: I had the chance earlier this week to get briefed by Rich Bowen, the Community Manager for CentOS Stream at Red Hat and get the inside track on what is really going on with CentOS Stream. Red Hat, and for that matter IBM, took a lot of heat with the announcement in September 2019 of the creation of CentOS Stream, and then things got worse for Red Hat when they made an announcement in December of 2020 that Red Hat’s sponsorship of CentOS Linux was changing.

Rich Bowen was very open and frank in our discussion and when I asked him how things are going eight months after the announcement, I got a very pragmatic answer. He shared that while most everyone agrees that CentOS stream is a good idea, people are still angry at how Red Hat made the announcements, as well as the lead time that the community was given. Rich even went further to describe his role this year as focused on rebuilding trust within the community.

Red Hat Endeavours to Rebuild Trust in CentOS

Trust is hard to build and really easy to lose. I very much got the impression that this sentiment is not lost on Red Hat and the CentOS community, given the events of the last eight months. I had the chance to chat with Rich on how he and the Red Hat team plans to rebuild the trust that has eroded since the announcement. It’s safe to say that one could easily describe the entirety of Rich’s approach as being focused on transparency. The inner workings of how open source projects work is not often discussed, as it is not newsworthy or, for that matter, very interesting. But is it vital work, and a lot of the governance that guides these various projects going on in boring process driven board meetings and through the drafting of bylaws provide both organization and clarity.

While open source communities are largely organic meritocracies where code rightly talks, somebody still has to clean out the sandbox every night and ensure clean sand is supplied regularly. My point is that the communities still need to have governance to function. If this governance is done well, projects flourish. If not then issues arise, factions form, and ultimately a project can wither and die as developers migrate elsewhere in protest.

I’m pleased to see the CentOS board is making solid efforts on a practical level to improve governance. These include small, but vital, steps such as making board meetings public and publishing these meetings on YouTube for all to see. While this may sound a very minor point, we have all seen the last 18-months the public’s reaction to decisions made behind closed doors that affect them. Meetings held in the open are the ethos of open source so, while this is a good step for the CentOS community, one has to ask why only now? So, ask that I did.

Rich’s response is that for some period of time, board had been on autopilot. With his joining the effort just over two years ago, he was keen to improve transparency and leverage his experience from his time at the Apache Foundation. The CentOS board is also changing in its makeup. Previously Red Hat had a stranglehold on the board with a bylaw stating that Red Hat must have the majority of board seats. This all changed in August, when the bylaws were rewritten to allow for a non-Red Hat majority, which will change the composition of decision making. And of course, it could also mean decisions go against the Red Hat interests going forward more so than they do today. This may sound deeply procedural, but board governance and the makeup of boards is, in my opinion, where transparency and trust are built in a community.

The board still has work to do to move away from being under undue influence of Red Hat. Board elections are slated for February and August 2022, but I believe positive first steps have been made. I will be watching closely to see how these elections go and whether the community is better represented going forward.

I asked Rich about whether he foresaw any plan to bring CentOS closer to the Linux Foundation Collaborative Projects’ structure and he did not, so we will have to see how the CentOS community moves forward in the longer term. In the short to medium term, I believe the focus on governance is well placed.

CentOS Stream Adoption

Given the relatively short period of time since the December 2020 announcement, I was keen to get a view of where developers are going and really get underneath what is really going on with CentOS Stream from a developer perspective. As is to be expected, a few loud voices in the community are still saying CentOS Stream is not what they want but, according to Rich, the numbers tell a different story. According to Red Hat, the numbers they are seeing show that CentOS stream is getting wider adoption than AlmaLinux and Rocky alternatives.

Bowen also highlighted how four significant Special Interest Groups (SIG) have come forward since the announcement in the Automotive, kmods, Infrastructure and Hyperscale SIGs while three of these SIGs are on the surface obvious the kmods SIG uilds kernel modules which are not available in either RHEL or base CentOS. This SIG provides an entry point for folks who want additional hardware, filesystems, or whatever, that Red Hat isn’t interested in. What I thought was interesting is that Facebook has a main board member now on the governing board in Davide Cavalca. It is worth noting here that board members act individually and do not officially represent their employer. As the hyperscale cloud providers get more and more into offering their own Linux distributions, I predict this trend will only continue and we will see more and more community involvement from the likes of AWS and the Microsoft Azure teams, and the continued heavy involvement from Google with Kubernetes and beyond.

One area where CentOS has alway sbeen strong has been in academia. I asked Rich about this and what he thinks is happening. He was quick to make clear that the likes of CERN and other large academic institutions have not moved away from CentOS, but he did say these institutions are reviewing their decisions. This will be key to the long term future of CentOS, as these institutions are a hotbed of innovation and provide a pipeline of students who turn into the enterprise developers of the future.

Red Hat has provided a program to entice the academic institutions feeling displaced by the transition to CentOS Streams and, according to Red Hat this program is gaining traction although I haven’t seen numbers. I will be interested to circle back on this topic with the Red Hat team in subsequent briefings.

What’s Next for CentOS Stream?

2021 has been a seminal year for CentOS stream, and certainly a lot of creative destruction has happened. My scorecard is that Red Hat have cleaned up the development flow from Fedora through CentOS Stream to RHEL. It is now clear where innovation, development and, ultimately, testing and support happens as code transitions through the various communities into Red Hat development proper. It is also clearer the options available for organizations and institutions to receive the support they need for RHEL, and the transition program from CentOS to RHEL at zero cost is to be applauded. This progress has, however, not come without conflict Rich and the various Red Hatters involved in the CentOS community still have a long way to go to fully rebuild trust, although their efforts so far this year have made a difference.

I envision more work to be done to make this transition be seen as fully successful and I fully expect some defection to AlmaLinux and Rocky as a result. Red Hat would have seen that as inevitable, but decided to move forward with the announcement regardless and, on reflection they have largely been proven right. CentOS Linux is a competitor to RHEL at the end of the day, so it is not surprising that Red Hat would want to have the competitor as much under control as was practicable.

What we saw here was, in my opinion, nothing more than bad communication and timing by Red Hat (and ultimately IBM). Given the community’s feelings toward IBM and the takeover of the beloved open source darling Red Hat by a big tech player, IBM should have delayed this announcement and given the community more time to digest the changes. However, given we are now eight months in, the noise has largely quieted down. Now, the hard work of rebuilding trust is underway and starting to show results.

Disclosure: Futurum Research is a research and advisory firm that engages or has engaged in research, analysis, and advisory services with many technology companies, including those mentioned in this article. The author does not hold any equity positions with any company mentioned in this article.