{"id":238,"date":"2015-08-12T13:04:26","date_gmt":"2015-08-12T11:04:26","guid":{"rendered":"http:\/\/www.hoekstra.nu\/wordpress\/?p=238"},"modified":"2020-04-12T06:44:03","modified_gmt":"2020-04-12T04:44:03","slug":"logstash-couchdb-elasticsearch-bending-your-rivers","status":"publish","type":"post","link":"https:\/\/www.hoekstra.nu\/wordpress\/blog\/2015\/08\/12\/logstash-couchdb-elasticsearch-bending-your-rivers\/","title":{"rendered":"LogStash \/ CouchDB \/ ElasticSearch -&gt; Bending your rivers"},"content":{"rendered":"<pre>ElasticSearch used to have a nice feature called \"<a href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/rivers\/current\/index.html\">rivers<\/a>\". \r\nUsing a river it was easy to get your CouchDB data into an ElasticSearch index.\r\nSince ElasticSearch 1.5 rivers are deprecated. (see <a href=\"https:\/\/www.elastic.co\/blog\/deprecating-rivers\">ElasticSearch rivers deprecated<\/a>)\r\n\r\nOne of the solutions is to use LogStash to ship your CouchDB data to ElasticSearch.\r\nIf you do so, you may encounter some small problems:\r\n1) When LogStash tries to delete a document in ElasticSearch, it does not work when you use HTTP protocol in the ElasticSearch output plugin.\r\n  -&gt; This is a bug. Use the node protocol instead. \r\n2) When LogStash tries to delete a document in ElasticSearch, the 'action' is wrong. It defaults to the action 'index', where it should be 'delete' for delete-actions.\r\n  -&gt; Modify your LogStash output definition for ElasticSearch as described below.\r\n3) When LogStash tries to delete a document in ElasticSearch, it needs the 'document_id'.\r\n  -&gt; Modify your LogStash output definition for ElasticSearch as described below.<\/pre>\n<pre>\r\ninput {\r\n\u00a0 couchdb_changes {\r\n\u00a0\u00a0\u00a0 host =&gt; \"localhost\"\r\n\u00a0\u00a0\u00a0 db\u00a0\u00a0 =&gt; \"my_db\"\r\n\u00a0\u00a0\u00a0 sequence_path =&gt; \".couchdb_my_db_seq\"\r\n\u00a0\u00a0\u00a0 type =&gt; \"my_type\"\r\n\u00a0\u00a0\u00a0 tags =&gt; [\"my_tag\"]\r\n\u00a0 }\r\n}\r\n\r\noutput {\r\n\u00a0 if \"my_tag\" in [tags] {\r\n\u00a0\u00a0\u00a0 if \"delete\" in [@metadata][action] {\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 elasticsearch {\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 action =&gt; \"delete\"\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 protocol =&gt; \"node\"\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 host =&gt; [\"localhost\"]\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 index =&gt; \"my_index\"\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 document_id =&gt; \"%{[@metadata][_id]}\"\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 manage_template =&gt; false\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 }\r\n\u00a0\u00a0\u00a0 } else {\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 elasticsearch {\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 action =&gt; \"index\"\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 protocol =&gt; \"node\"\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 host =&gt; [\"localhost\"]\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 index =&gt; \"my_index\"\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 document_id =&gt; \"%{[@metadata][_id]}\"\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 manage_template =&gt; false\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 }\r\n\u00a0\u00a0\u00a0 }\r\n\u00a0 }\r\n}\r\n\r\n<\/pre>\n","protected":false},"excerpt":{"rendered":"<p>ElasticSearch used to have a nice feature called &#8220;rivers&#8221;. Using a river it was easy to get your CouchDB data into an ElasticSearch index. Since ElasticSearch 1.5 rivers are deprecated. (see ElasticSearch rivers deprecated) One of the solutions is to use LogStash to ship your CouchDB data to ElasticSearch. If you do so, you <span style=\"color:#777\"> . . . &rarr; Read More: <a href=\"https:\/\/www.hoekstra.nu\/wordpress\/blog\/2015\/08\/12\/logstash-couchdb-elasticsearch-bending-your-rivers\/\">LogStash \/ CouchDB \/ ElasticSearch -&gt; Bending your rivers<\/a><\/span><\/p>\n","protected":false},"author":96,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[3],"tags":[],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/www.hoekstra.nu\/wordpress\/wp-json\/wp\/v2\/posts\/238"}],"collection":[{"href":"https:\/\/www.hoekstra.nu\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.hoekstra.nu\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.hoekstra.nu\/wordpress\/wp-json\/wp\/v2\/users\/96"}],"replies":[{"embeddable":true,"href":"https:\/\/www.hoekstra.nu\/wordpress\/wp-json\/wp\/v2\/comments?post=238"}],"version-history":[{"count":2,"href":"https:\/\/www.hoekstra.nu\/wordpress\/wp-json\/wp\/v2\/posts\/238\/revisions"}],"predecessor-version":[{"id":240,"href":"https:\/\/www.hoekstra.nu\/wordpress\/wp-json\/wp\/v2\/posts\/238\/revisions\/240"}],"wp:attachment":[{"href":"https:\/\/www.hoekstra.nu\/wordpress\/wp-json\/wp\/v2\/media?parent=238"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.hoekstra.nu\/wordpress\/wp-json\/wp\/v2\/categories?post=238"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.hoekstra.nu\/wordpress\/wp-json\/wp\/v2\/tags?post=238"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}