Puppet Without MastersCurrent Working Directoryhttp://current.workingdirectory.net/posts/2011/puppet-without-masters/Current Working Directoryikiwiki2012-12-07T14:26:35Zcomment 1http://current.workingdirectory.net/posts/2011/puppet-without-masters/comment_1_249aa16325840d29956b56d38cf35bbe/Anonymous2011-05-31T14:58:06Z2011-05-31T14:58:06Z
Very interesting read. Bookmarked for future reference. Thank you.
Bcfg2http://current.workingdirectory.net/posts/2011/puppet-without-masters/comment_2_6a0614d3f3517dbd20dde5cf284763c3/Anonymous2011-05-31T15:17:02Z2011-05-31T15:17:02Z
You might also want to take a look at Bcfg2 (http://docs.bcfg2.org/). It uses a different mindset than many of its contemporaries so you may find that it suits you better. The IRC channel (and the community in general) is extremely helpful with any questions you might have.
not an argument for mastershttp://current.workingdirectory.net/posts/2011/puppet-without-masters/comment_3_e408f3bad7fa352884d49ab3b06a1bcc/micah [id.mayfirst.org]2011-06-01T14:23:22Z2011-06-01T14:23:21Z
<p>Thanks for the great write-up about masterless setups! I was eager to hear how this was going, and its good to see that it is going well.</p>
<p>There are a few things in your post that I wanted to comment on. To be clear, I'm not making an argument for a master setup, I actually think that a masterless setup is the way to go in a lot of ways, however I dont agree with you about some points.</p>
<blockquote><p>To make matters worse, it seems as though a common practice is to generate and store all server ssh keys (private and public) on the puppet master and then push the private keys to their respective nodes. That means an intruder doesn't need to write to the puppet master, just reading these keys would be enough to compromise all servers in your network.</p></blockquote>
<p>I'm not sure that this is a common practice, I'm curious where you got that impression from?</p>
<blockquote><p>(I also learned more reasons for going without a puppet master, like not needing a server with 16GB of RAM!)</p></blockquote>
<p>I'm dont know where you got 16GB from (perhaps from someone who is running a very old version of puppet, which did have some memory issues?). But even with the memory issues, 16GB is more than I've ever needed! I know people who are running puppetmaster with only 256megs of RAM... that said, scaling puppetmaster is a known issue in the community, but I dont think it is as drastic as you portray.</p>
<blockquote><p>It's possible to run puppet with storeconfigs but without running a puppet master (that gets around the bugginess and resource consumption of the puppetmaster and puppet daemons, while providing the convenience of centralization). For our purposes, however, we decided we did not want any form of centralization that would provide an additional point of vulnerability.</p></blockquote>
<p>I am also not really convinced that storeconfigs presents a point of vulnerability, or that going masterless eliminates one. My storeconfigs database holds pretty trivial, non-compromising information that just links a hostname to a resource, such as nagios. It is a centralized resource, so by definition there is a general vulnerability there, but that isn't specific to storedconfigs. In fact with a masterless setup there are other vulnerabilities that you are getting, that you wouldn't otherwise have with a puppetmaster setup. For example, every masterless node has write access to the storedconfigs, which allows any compromised node to inject files on any other node that is doing file collection.</p>
<blockquote><p>There's no question - the setup was rather tedious (we're using runit to maintain an ssh-agent for each root user)</p></blockquote>
<p>I have the impression from reading this that the only tedious thing that you ran into was using runit to maintain a ssh-agent for each root user, but I'm guessing there are other tediums involved, and I'm interested to know what those are. I suspect that your post takes the approach of highlighting the disadvantages of doing a puppetmaster setup, and downplaying the pain in running a masterless setup. I think that there is a lot more pain than you have detailed, which is the part I am interested in.</p>
<p>I'm also not really sure I understand what the purpose the ssh-agent serves in this setup?</p>
<blockquote><p>Shared modules</p></blockquote>
<p>Your discussion of shared modules is confusing to me in a number of respects. First of all, I know it is possible to use shared modules on masterless nodes, so I dont see this as an argument either for a masterless puppet setup, or against shared modules.</p>
<blockquote><p>Now enter the puppet module. In addition to learning puppet syntax (and struggling with git) you now need to understand how the third party module works. With software programming, I typically don't need or want to learn how a library or class does what it does - that's the beauty of object-oriented programming: it hides the complexity. But when it comes to configuring the servers that I will be responsible for debugging and maintaining, I really need to know exactly what is happening.</p></blockquote>
<p>I guess I fundamentally disagree with you here. I think learning how a shared module works is actually quite a useful process. Its a great educational opportunity to learning things about puppet from other people, and I think it pays off in the long run, enormously. I guess I dont think a shared module like an abstraction like a library, which may be why I think differently about them, I dont just take a module and throw it down without understanding how it works, or what it does, in fact until I am comfortable with the module doing what I need, I dont use it. I find them super useful, the network effects gained from collaborative efforts vastly outweighs the time it takes to understand what the module is doing.</p>
<p>That said, I fully understand the learning curve involved in puppet, and can understand the argument that learning a module is another thing that must be overcome. However, I dont think that means discounting shared modules, rather it just means you aren't ready to take on that additional burden yet, but at some point your familiarity with puppet will make the shared module learning curve flatten out and instead of it being a burden, the benefits will be clear.</p>
<blockquote><p>To further compound the problem, I found myself wading through third party module code designed to work on Debian, Ubuntu, CentOS, Redhat, gentoo... and more. We run entirely on Debian - we don't need any of this extra code. And, once I got rid of all the other operating systems, I was still left with a complex module that allows you to configure software in ways we'll never need.</p></blockquote>
<p>I dont find this as problematic as you do, its actually quite easy to ignore the other operating systems, and the modules aren't as complicated as I feel you are making them out to be. Finally, not taking advantage of all the possible ways to configure software is not a bad thing in my opinion. Especially when later I find the need for those things that I didn't need before. In fact, most software I use has functionality that I never need (eg. aptitude moo).</p>
<blockquote><p>In the end, we tore out most of these third party modules and replaced them with file and exec puppet resources that did exactly what we needed them to do. Our code base is now much smaller and simpler.</p></blockquote>
<p>My understanding was you switched to puppet to get away from writing bash scripts, this sounds like you are just using puppet to write bash scripts. This is where your comment about libraries belongs, puppet provides you with abstracted types, to hide complexity, its better to use those! I will certainly admit that its not always easy to find a way to do that, and I often recommend that people who are getting going with puppet start simply by just shipping the configuration file and some execs, but it is often said in the puppet communities that overuse of file and exec resources is an indication that something is not right. I think its a little more nuanced than this, but essentially true.</p>
<p>I don't really see the argument about smaller being something that is a benefit, compared to what you lose. Even the most complicated module that I've seen that has tests, and configuration files is only a few hundred K, which is nothing.</p>
<p>Finally, the shared module discussion doesn't seem to be related to a masterless setup at all, it seems more of a rant about your frustration with shared modules (ie. unreadable, and multi-distro). There are plenty of modules that do not use storedconfigs, and work fine with a masterless setup, and personally, I would love to see any issues you ran into with shared modules and a masterless setup be fed back to the shared-module community, so others can benefit from the frustrating efforts you have been going through. I'd love to be able to switch to a masterless setup some day, and having that capability built into modules would make that all the more easier.</p>
re: not an argument for mastershttp://current.workingdirectory.net/posts/2011/puppet-without-masters/comment_4_ef7fc8f925dcef8944972e6ff5fc9e08/jamie [id.mayfirst.org]2011-07-01T02:43:07Z2011-06-02T14:51:29Z
<blockquote><p>Thanks for the great write-up about masterless setups! I was eager to hear how this was going, and its good to see that it is going well.</p></blockquote>
<p>Thank you for the thoughtful responses :).</p>
<blockquote><blockquote><p>To make matters worse, it seems as though a common practice is to generate and store all server ssh keys (private and public) on the puppet master and then push the private keys to their respective nodes. That means an intruder doesn't need to write to the puppet master, just reading these keys would be enough to compromise all servers in your network.</p></blockquote>
<p>I'm not sure that this is a common practice, I'm curious where you got that impression from?</p></blockquote>
<p>I got the impression from the ssh_keygen function in the <a href="https://labs.riseup.net/code/projects/module-sshd">sshd shared puppet module</a>. I was trying to use it without success (I thought the funciton was run on the node). I asked in irc.indymedia.org#puppet and (sorry, don't remember who) explained that functions are always run on the puppetmaster. They went on to explain that they push all keys from the puppetmaster to the nodes after creation, keeping a backup on the puppetmaster.</p>
<p>I made a pretty big and unsubstantiated logical leap in my blog (now corrected) about this being a common practice. I really don't have enough experience to say one way or another.</p>
<blockquote><blockquote><p>(I also learned more reasons for going without a puppet master, like not needing a server with 16GB of RAM!)</p></blockquote>
<p>I'm dont know where you got 16GB from (perhaps from someone who is running a very old version of puppet, which did have some memory issues?). But even with the memory issues, 16GB is more than I've ever needed! I know people who are running puppetmaster with only 256megs of RAM... that said, scaling puppetmaster is a known issue in the community, but I dont think it is as drastic as you portray.</p></blockquote>
<p>Yes - you are right again. I know I found someone reporting the 16GB of number, but can't seem to find it. Sloppy of me. I've just removed that reference as well.</p>
<blockquote><blockquote><p> It's possible to run puppet with storeconfigs but without running a puppet master (that gets around the bugginess and resource consumption of the puppetmaster and puppet daemons, while providing the convenience of centralization). For our purposes, however, we decided we did not want any form of centralization that would provide an additional point of vulnerability.</p></blockquote>
<p>I am also not really convinced that storeconfigs presents a point of vulnerability, or that going masterless eliminates one. My storeconfigs database holds pretty trivial, non-compromising information that just links a hostname to a resource, such as nagios. It is a centralized resource, so by definition there is a general vulnerability there, but that isn't specific to storedconfigs. In fact with a masterless setup there are other vulnerabilities that you are getting, that you wouldn't otherwise have with a puppetmaster setup. For example, every masterless node has write access to the storedconfigs, which allows any compromised node to inject files on any other node that is doing file collection.</p></blockquote>
<p>I think we're on the same page here - if your goal is to reduce single points of vulnerabilities, then it doesn't make sense to run puppet without a master but with storeconfigs. My mention of that option is only in passing. I just did a minimal update to try to clarify that point - we are running neither the master or storeconfigs.</p>
<blockquote><blockquote><p>There's no question - the setup was rather tedious (we're using runit to maintain an ssh-agent for each root user)</p></blockquote>
<p>I have the impression from reading this that the only tedious thing that you ran into was using runit to maintain a ssh-agent for each root user, but I'm guessing there are other tediums involved, and I'm interested to know what those are. I suspect that your post takes the approach of highlighting the disadvantages of doing a puppetmaster setup, and downplaying the pain in running a masterless setup. I think that there is a lot more pain than you have detailed, which is the part I am interested in.</p></blockquote>
<p>Ha. Well, maybe it's all in the phrasing. This blog covers in detail all the problems we had - just phrased in terms of the solutions. Every point of difference described in this post led to hours and even days of research, extra fanangling and heart ache to get it working the way we wanted it to work.</p>
<p>To the credit of puppet (and monkeysphere), I don't think we had to make any compromises (in terms of the functionality we wanted) to run puppet without a master. Others, who rely on reporting mechanisms (or any of the many features of puppet that I still don't know about) that are provided by storeconfigs or puppetmaster may not have the same experience.</p>
<p>The main compromise was in the time it took to set things up. I probably spent 3 - 4 times the number of hours to get us up and running that it would have taken if we used a more traditional model.</p>
<blockquote><p>I'm also not really sure I understand what the purpose the ssh-agent serves in this setup?</p></blockquote>
<p>Again - more tangential than anything else. Since each root user has to ssh, using the monkeysphere, to the backup servers, it needs to have ssh-agent running to handle the monkeysphere authentication.</p>
<blockquote><blockquote><p> Shared modules</p></blockquote>
<p>Your discussion of shared modules is confusing to me in a number of respects. First of all, I know it is possible to use shared modules on masterless nodes, so I dont see this as an argument either for a masterless puppet setup, or against shared modules.</p></blockquote>
<p>Yes - that is correct - shared modules work great in a master-less setup. Perhaps the blog is mis-titled. Masterless puppet is the main focus, but while I was at it, I thought I would explore more of the differences in our approach from the standard approach.</p>
<blockquote><blockquote><p> Now enter the puppet module. In addition to learning puppet syntax (and struggling with git) you now need to understand how the third party module works. With software programming, I typically don't need or want to learn how a library or class does what it does - that's the beauty of object-oriented programming: it hides the complexity. But when it comes to configuring the servers that I will be responsible for debugging and maintaining, I really need to know exactly what is happening.</p></blockquote>
<p>I guess I fundamentally disagree with you here. I think learning how a shared module works is actually quite a useful process. Its a great educational opportunity to learning things about puppet from other people, and I think it pays off in the long run, enormously. I guess I dont think a shared module like an abstraction like a library, which may be why I think differently about them, I dont just take a module and throw it down without understanding how it works, or what it does, in fact until I am comfortable with the module doing what I need, I dont use it. I find them super useful, the network effects gained from collaborative efforts vastly outweighs the time it takes to understand what the module is doing.</p></blockquote>
<p>I have also learned enormously about puppet from reading other people's shared modules. It's probably the single best way to learn how to write your own puppet code.</p>
<blockquote><p>That said, I fully understand the learning curve involved in puppet, and can understand the argument that learning a module is another thing that must be overcome. However, I dont think that means discounting shared modules, rather it just means you aren't ready to take on that additional burden yet, but at some point your familiarity with puppet will make the shared module learning curve flatten out and instead of it being a burden, the benefits will be clear.</p></blockquote>
<p>This blog represents version .01 of MFPL's puppet repository :). We'll see how it changes. I'm certainly re-acting in part to the MFPL support meeting when I presented version .001 (with dozens of shared modules and the need to use git-submodules). I almost got thrown out the window.</p>
<blockquote><blockquote><p>To further compound the problem, I found myself wading through third party module code designed to work on Debian, Ubuntu, CentOS, Redhat, gentoo... and more. We run entirely on Debian - we don't need any of this extra code. And, once I got rid of all the other operating systems, I was still left with a complex module that allows you to configure software in ways we'll never need.</p></blockquote>
<p>I dont find this as problematic as you do, its actually quite easy to ignore the other operating systems, and the modules aren't as complicated as I feel you are making them out to be. Finally, not taking advantage of all the possible ways to configure software is not a bad thing in my opinion. Especially when later I find the need for those things that I didn't need before. In fact, most software I use has functionality that I never need (eg. aptitude moo).</p>
<blockquote><p> In the end, we tore out most of these third party modules and replaced them with file and exec puppet resources that did exactly what we needed them to do. Our code base is now much smaller and simpler.</p></blockquote>
<p>My understanding was you switched to puppet to get away from writing bash scripts, this sounds like you are just using puppet to write bash scripts. This is where your comment about libraries belongs, puppet provides you with abstracted types, to hide complexity, its better to use those! I will certainly admit that its not always easy to find a way to do that, and I often recommend that people who are getting going with puppet start simply by just shipping the configuration file and some execs, but it is often said in the puppet communities that overuse of file and exec resources is an indication that something is not right. I think its a little more nuanced than this, but essentially true.</p>
<p>I don't really see the argument about smaller being something that is a benefit, compared to what you lose. Even the most complicated module that I've seen that has tests, and configuration files is only a few hundred K, which is nothing.</p>
<p>Finally, the shared module discussion doesn't seem to be related to a masterless setup at all, it seems more of a rant about your frustration with shared modules (ie. unreadable, and multi-distro). There are plenty of modules that do not use storedconfigs, and work fine with a masterless setup, and personally, I would love to see any issues you ran into with shared modules and a masterless setup be fed back to the shared-module community, so others can benefit from the frustrating efforts you have been going through. I'd love to be able to switch to a masterless setup some day, and having that capability built into modules would make that all the more easier.</p></blockquote>
<p>Our puppet setup is definitely an early work in progress - and your comments reflect the perspective of someone with many more years of puppet experience under their belt.</p>
<p>You have prompted me to finally <a href="https://projects.puppetlabs.com/issues/7759">submit the one puppet bug that inhibits the use of shared modules between sites using storeconfigs and those not using storeconfigs</a>. Let's hope it gets fixed!</p>
Cfengine 3http://current.workingdirectory.net/posts/2011/puppet-without-masters/comment_5_62c5697c247bb8a4bd7c98f55e17d212/Anonymous2011-07-01T02:43:07Z2011-06-05T04:54:20Z
<p>Hi,</p>
<p>Interesting read.
However, if you are not too "religious" about using Puppet, I would suggest checking out Cfengine 3 (www.cfengine.org).
It has no requirement of "masters" (or even network connectivity) whatsoever - it is totally flexible on how you architect it.
Puppet is based on ideas of Cfengine 2, but Cfengine 3 came in 2008 and is my favorite at the moment because of its unparalleled power and flexibility.</p>
<p>John</p>
Created additional complexity http://current.workingdirectory.net/posts/2011/puppet-without-masters/comment_6_02cb64e7428178ecb6bd7644baa23010/Anonymous2012-12-07T14:21:19Z2012-12-07T07:06:18Z
<p>So you created additional complexity of your system by replacing the centralization of a puppetmaster that you own and control with the centralization of a github repository that you do not own, nor control and have to pay a subscription for?</p>
<p>Is this because you don't like the word master?</p>
<p>Not a win in my book.</p>
Re: Created additional complexityhttp://current.workingdirectory.net/posts/2011/puppet-without-masters/comment_7_jamie/jamie2012-12-07T14:26:35Z2012-12-07T09:17:18Z
<p>My experience suggests that most people who run a puppetmaster also keep their puppetmaster files in git. So, we're actually reducing complexity by removing the puppetmaster from the equation, not replacing it with something new. And I couldn't agree more about github - why would anyone do that? We run our own git repository (all it takes is ssh and some disk space).</p>
<p>If you re-read my blog you'll see that I like the use of the term puppetmaster - it's technology design that I don't like.</p>