{"id":190,"date":"2013-02-15T07:10:39","date_gmt":"2013-02-15T06:10:39","guid":{"rendered":"http:\/\/www.hoekstra.nu\/wordpress\/?p=190"},"modified":"2020-04-12T06:43:47","modified_gmt":"2020-04-12T04:43:47","slug":"puppet-couchdb-a-powerful-combination","status":"publish","type":"post","link":"https:\/\/www.hoekstra.nu\/wordpress\/blog\/2013\/02\/15\/puppet-couchdb-a-powerful-combination\/","title":{"rendered":"Puppet, Puppet functions &amp; CouchDB: A powerful combination"},"content":{"rendered":"<p>Are you automating your deployments with Puppet?<br \/>\nWe are too.I believe it&#8217;s a great tool to manage your infrastructure in an organized way.<\/p>\n<p>But what about all the rich data that is stored in the Puppet nodes files? Isn&#8217;t that data that could be useful for other processes\/purposes?<br \/>\nYes it is, and that&#8217;s why we selected a database to store all this CMDB information: CouchDB<\/p>\n<h1><\/h1>\n<h1>Why do this?<\/h1>\n<h2>Historical:<\/h2>\n<p>At the time that we started separating our data\/information from our Puppet recipes Hiera was a very young product (or did not exist yet, I don&#8217;t remember exactly).<\/p>\n<h2>A single source of truth:<\/h2>\n<p>We wanted one source for all the objects we are managing. If all scripts and tools read from one data-source, then there is no other thruth. So Puppet had to comply with this.<\/p>\n<h2>Openness:<\/h2>\n<p>I don&#8217;t know if Hiera could do this, but we need our CMDB to be as open and flexible as possible. Our CMDB is a very rich information source, and a lot of other tools\/scripts require access to this data in a light-weight way.<\/p>\n<h2>Information variety:<\/h2>\n<p>Our CouchDB CMDB contains a lot more information than just Puppet configurations. We register all devices (nodes), configurations and services we deliver to customers in our CouchDB CMDB now.<br \/>\nIn the future we also intend to store service-levels, service-windows, change-windows, service-owner-information, maintainer-information, etc&#8230;<\/p>\n<p>Our CouchDB CMDB also contains all relations between these objects:<\/p>\n<ul>\n<li>horizontal relations\n<ul>\n<li>TCP connections (from loadbalancer to web-proxy, from web-proxy to webserver, from webserver to database, etc)<\/li>\n<\/ul>\n<\/li>\n<li>vertical (hierarchical) relations\n<ul>\n<li>Availability relations: If this object goes down, what would be the impact?<\/li>\n<li>Maintainability relations: Which objects are needed to maintain the service, but do not cause an Incident when they are down?<\/li>\n<li>Financial relations: How does the cost of a device contribute to the infrastructure cost of a service?<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h2>Separated responsibilities:<\/h2>\n<p>We also register objects that are not managed\/deployed by Puppet. Even objects that are managed by other teams and companies.Why? Because we want to be able to calculate the impact if one of these configurations goes down.<\/p>\n<h1><\/h1>\n<h1>But why CouchDB?<\/h1>\n<ul>\n<li>Reading from CouchDB is fast<\/li>\n<li>You don&#8217;t need a driver to access your data: A simple HTTP GET request does the trick<\/li>\n<li>It is JSON-based, which makes the data accessible for almost every software-language \/ script<\/li>\n<li>The map\/reduce mechanism allows you to quickly count\/sum data to generate reports.<\/li>\n<\/ul>\n<h1><\/h1>\n<h1>How we did this?<\/h1>\n<p>Using one simple Puppet function.<br \/>\nIt does a CouchDB lookup for a configuration, and gets a JSON object in return.<br \/>\nThe Puppet function parses the JSON to Ruby objects, and delivers it to the Puppet manifest. (Ruby DSL)<\/p>\n<p>One small drawback:<\/p>\n<ul>\n<li>You really need to check the data that is returned from the CouchDB lookup.<br \/>\nThe CouchDB document can contain all sorts of key\/values, and you need to make sure that this data is valid before you continue executing the second part of your manifest.<br \/>\nAt least check if the obligatory keys are present.<br \/>\nIf you don&#8217;t check the CouchDB data, and the data is invalid, then you may get some nasty Puppet errors in your Puppet agent log, which are hard to understand.<\/li>\n<\/ul>\n<p>Here is the Puppet function we created:<\/p>\n<p><a title=\"http:\/\/www.kastelijn.nu\/configuration_lookup.rb\" href=\"http:\/\/www.kastelijn.nu\/configuration_lookup.rb\">http:\/\/www.kastelijn.nu\/configuration_lookup.rb<\/a><\/p>\n<p>Feel free to use it, give feedback, etc..<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Are you automating your deployments with Puppet? We are too.I believe it&#8217;s a great tool to manage your infrastructure in an organized way.<\/p>\n<p>But what about all the rich data that is stored in the Puppet nodes files? Isn&#8217;t that data that could be useful for other processes\/purposes? Yes it is, and that&#8217;s why <span style=\"color:#777\"> . . . &rarr; Read More: <a href=\"https:\/\/www.hoekstra.nu\/wordpress\/blog\/2013\/02\/15\/puppet-couchdb-a-powerful-combination\/\">Puppet, Puppet functions &amp; CouchDB: A powerful combination<\/a><\/span><\/p>\n","protected":false},"author":96,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[3,1],"tags":[68,64,66,70,65,67,69],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/www.hoekstra.nu\/wordpress\/wp-json\/wp\/v2\/posts\/190"}],"collection":[{"href":"https:\/\/www.hoekstra.nu\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.hoekstra.nu\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.hoekstra.nu\/wordpress\/wp-json\/wp\/v2\/users\/96"}],"replies":[{"embeddable":true,"href":"https:\/\/www.hoekstra.nu\/wordpress\/wp-json\/wp\/v2\/comments?post=190"}],"version-history":[{"count":22,"href":"https:\/\/www.hoekstra.nu\/wordpress\/wp-json\/wp\/v2\/posts\/190\/revisions"}],"predecessor-version":[{"id":267,"href":"https:\/\/www.hoekstra.nu\/wordpress\/wp-json\/wp\/v2\/posts\/190\/revisions\/267"}],"wp:attachment":[{"href":"https:\/\/www.hoekstra.nu\/wordpress\/wp-json\/wp\/v2\/media?parent=190"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.hoekstra.nu\/wordpress\/wp-json\/wp\/v2\/categories?post=190"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.hoekstra.nu\/wordpress\/wp-json\/wp\/v2\/tags?post=190"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}