System Administration Automation

System Administration
 
Standing Room Only – 30 people. 
Adam talked about Puppet and the plans to move to puppet 3. Bill McAllister asked what configuration management tools were being used besides puppet and Booker (SLAC) has their config management tool called Taylor and this was written in house. It’s heavily dependent on AFS and is written in perl.  You can write a command and it will create a temp file and do comparisons and if the change isn’t needed it skips it. One of the big advantages of Puppet and Chef is that there is a library of tools that are out there.  With Taylor, everything you need you need to build.  SLAC would like to move to Chef. 
 
People wondered how often we need write our own modules and how often is there something available out on Puppet Labs.  Adam said that more often than not they are writing the modules.   They wrote modules for SSH and Wallet and while there might have been some out there, they still wrote their own.  School of Medicine just went live on Puppet3.  For Centos, it’s been grabbing what’s available and make slight modifications.  They have had to make some changes for their patching process.  More often than not they start with the community modules and tweak things from there. 
 
Bill spoke about the splitting of the puppet manifests without losing the shared configurations.  Bill mentioned Adam wrote something to handle this. Bill mentioned once you get on a configuration management system, there’s a lot to think about in each upgrade. SoM took about a year to move to v3. 
A poll was taken about Puppet, Chef and Salt. 
 
SLAC has been looking at things and there was a large project where they created a sandbox environment for Puppet and Chef where people could test and make a decision as a group and they weren’t successful because no one has the time to do the analysis. To Booker the big advantage of Chef is that the database is built in and they need that database so that was the key deciding point and he doesn’t care for the puppet dsl.  Chef has been selected but getting there is a whole other matter. Carving out time to make the transition hasn’t happened.  There were a couple of transition schemes that were discussed but they haven’t moved forward.  The current plan is to transition over time and use Chef to manage RHEL7 and beyond. Context switching between two different tools is going to be difficult.
 
The problem with management tools is that once you invest in one and build your processes and practices into that management tool, you are pretty much stuck with that tool and moving off the tool is quite difficult. One advantage is that you no longer need to maintain documentation because the information is in puppet.  Adam spoke about the challenge of working with applications people who lock puppet because people can lock puppet and it defeats the purpose of having puppet. 
 
The problem with Taylor (at SLAC) is that it has a bad reputation because when something goes wrong, people want to blame it. Even producing the black box in a way that people care about it.  Maybe with Puppet or Chef this will be easier. All of these tools are powerful and you need to be careful not to shoot yourself in your foot. 
 
Someone asked how to share modules between groups. Adam discussed a small application he wrote to pull in changes from Git branch/tag/revision and it syncs. Adam talked about Puppet Librarian which is similar which people can use. They tried using Git submodules and that was a disaster. Someone asked who to separate core and custom department modules. Adam mentioned there are shared modules people can use, then there are local modules that are written internally and they are in the group repository. Everyone has separate Git repositories like ISO, Application Support, etc.  This is using puppet environments to configure Git branches. Not using Git or svn would be crazy. Individual repositories have a stable branch and a test branch. Getting to the point of handling this was complicated and was an effort. Someone also mentioned puppet is second to your source code management tool. Riel mentioned they wanted to move to Git from svn. 
 
Adam is interested in hearing from groups that are not using configuration management. Leroy talked about Linux and Windows and they use a configuration mgmt tool for Linux but need one for Windows.  Booker talked about Windows and PowerShell and that Microsoft has jumped on board with this for desired state configuration.  That gives a hook for puppet and chef to treat the configuration of Microsoft machine as resources.  He is pretty sure this is in the latest version of puppet and chef client to do this.  Now you can access they desired state configuration API with your tool. It’s the early days but there is desire and momentum in the community. For puppet it’s version 3 and up. 
 
Earth Sciences is in the position of adopting puppet for 10 systems. They are interested as it includes the documentation.  They originally looked at Puppet and they felt it seemed too big and complicated for their needs. They looked at Chef, Salt and Ansible and they think that are going to move forward with this. There are only 2 people managing 10 machines and apps so they are interested in knowing feedback on Ansible. Booker played around with Ansible and thinks this is a good choice for the Earth Sciences scale organization.  It’s important to remember that once you invest in something you’re not going to move to something else but choose wisely.  Adam chimed in that the people at Stanford Online is using this and he encouraged reaching out to them for information.
 
ACS really invests in making builds reproducible. They generate packages for pretty much everything that gets put on the machine. Riel mentioned that they also do a lot of packaging and maintain a repo. In Computer Science they do repackaging and they want to have it nicely contained. To manage drift we try to stick as close to possible to the core and the modifications are tiny or vanilla if possible. 
Does anyone use Cloud AWS or Azure with puppet? Riel has started on a couple and is using Elastic Beanstalk and that has its own configuration handler on that.  The www servers are running this.  Xueshan Feng is the one who developed this. 
 
Someone asked about moving away from Puppet to another tool and people weighed in that just going from 2.7 to 3.0 was a challenge, so unless Puppet was failing you, why would you move? Chef = Ruby, Salt = Python and there was some commentary on comfort level of these. If you’ve already gone down a path, what compelling reason would one have to switch? It’s something you need to do right the first time because one of the deciding factors are the communities that come with them – which is why SLAC is moving from their home grown tool to a more broadly used tool – because with it comes the community. There is now a good puppet community. 
Adam was hoping there was a puppet community of practice and others spoke about it. Puppet-users@lists.stanford.edu is the mailing list for puppet. 
Office of Development is also looking to move to puppet (or is already using it). 
Adam mentioned that the ACS group, ISO, CRC that has a Linux coordination meeting.  Any folks who are interested in attending this are welcome to attend to collaborate cross group more. 
Booker asked how people are testing. Adam has written tests for his puppet code and it didn’t work so well. Riel said they use some sandboxes to roll their tests to for testing. Someone else uses Git Environments to test – they are doing tests by experimentation. They have no formal test framework. ACS has a dashboard that includes puppet, tripwire etc. 
Adam went to puppet conf and one thing that was different for him was that they push really hard the things that have a revenue stream – like puppet enterprise which has dashboards etc. ACS had a contract but they ditched it because they weren’t using it. Puppet Dashboard is open source, but Puppet Enterprise is a for fee module.