Your IP Address is:

Next Fedora Release

LogStash / CouchDB / ElasticSearch -> Bending your rivers

ElasticSearch used to have a nice feature called "rivers". 
Using a river it was easy to get your CouchDB data into an ElasticSearch index.
Since ElasticSearch 1.5 rivers are deprecated. (see ElasticSearch rivers deprecated)

One of the solutions is to use LogStash to ship your CouchDB data to ElasticSearch.
If you do so, you may encounter some small problems:
1) When LogStash tries to delete a document in ElasticSearch, it does not work when you use HTTP protocol in the ElasticSearch output plugin.
  -> This is a bug. Use the node protocol instead. 
2) When LogStash tries to delete a document in ElasticSearch, the 'action' is wrong. It defaults to the action 'index', where it should be 'delete' for delete-actions.
  -> Modify your LogStash output definition for ElasticSearch as described below.
3) When LogStash tries to delete a document in ElasticSearch, it needs the 'document_id'.
  -> Modify your LogStash output definition for ElasticSearch as described below.
input {
  couchdb_changes {
    host => "localhost"
    db   => "my_db"
    sequence_path => ".couchdb_my_db_seq"
    type => "my_type"
    tags => ["my_tag"]
  }
}

output {
  if "my_tag" in [tags] {
    if "delete" in [@metadata][action] {
      elasticsearch {
        action => "delete"
        protocol => "node"
        host => ["localhost"]
        index => "my_index"
        document_id => "%{[@metadata][_id]}"
        manage_template => false
      }
    } else {
      elasticsearch {
        action => "index"
        protocol => "node"
        host => ["localhost"]
        index => "my_index"
        document_id => "%{[@metadata][_id]}"
        manage_template => false
      }
    }
  }
}

Comments are closed.