prune removes @timestamp and crashes pipeline

[marnikvde]

Using prune like this made the pipeline crash:

prune { whitelist_names => [ "^a$", "^b$", "^c$", "^d$"] }

Apparently the problem is that the @timestamp field gets removed:

logstash_1 | [2018-02-16T23:06:56,656][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Error: timestamp field is missing>, :backtrace=>["org/logstash/ext/JrubyEventExtLibrary.java:168:in `sprintf'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:169:in `event_action_params'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:39:in `event_action_tuple'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:34:in `block in multi_receive'", "org/jruby/RubyArray.java:2486:in `map'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:34:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:13:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:50:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:487:in `block in output_batch'", "org/jruby/RubyHash.java:1343:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:486:in `output_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:438:in `worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:393:in `block in start_workers'"]} dockerelk_logstash_1 exited with code 1

Not sure if that is intended. If so, it's probably good to mention that in the documentation.

(about to test what happens to the [@metadata][...] fields now)

[magnusbaeck]

This is definitely not intended. The @timestamp field is special so we'll have to skip it implicitly. It's not reasonable that users will have to explicitly whitelist that field.

[asantas93]

I am also experiencing this issue. When I whitelist @timestamp though, the field gets written to elasticsearch as if it's a normal field. Is there another workaround?

[magnusbaeck]

When I whitelist @timestamp though, the field gets written to elasticsearch as if it's a normal field.

What do you mean? It becomes a string rather than a date? That's unrelated to problem.

[asantas93]

I mean that a key @timestamp shows up under _source when I view the document in elasticsearch. Is that expected?

[w74]

Any news on this? I too, am seeing this error in my pipelines.

[fozboz]

Unable to reproduce @asantas93 issue with these filters. @timestamp is in the correct place, but I must explicitly whitelist it.

filter {
  json {
    source => "message"
  }
  if [@metadata][kafka][topic] == "resource-events" {
    date {
      match => [ "[content][created]", "UNIX_MS" ]
    }
    clone {
      clones => [ "cloned" ] # Clone the event
      add_field => { "[@metadata][cloned]" => true }
    }
    if [@metadata][cloned] == "true" { # If this is the cloned event
      mutate {
        copy => { "[content][data][results][files][binary]" => "file" } # Required workaround for prune filter
      }
      prune {
        whitelist_names => [ "^file$", "^\@timestamp$" ] # Only keep the binary file list and the timestamp
      }
      split { # Split out the array so we generate individual events for each file
        field => "file"
        add_field => { "[@metadata][id]" => "%{[file][file-created]}" } # Use the file ID in ES
        add_tag => [ "file" ]
      }
    } else { # This isn't the cloned event
      mutate {
        add_field => { "[@metadata][id]" => "%{[content][id]}" } # use the event ID in ES
      }
    }
  } else { # Every other event, errors etc.
    fingerprint {
      source => "message"
      target => "[@metadata][id]" # Generate an ID
      method => "MD5"
    }
    date {
      match => [ "kafka-timestamp", "UNIX_MS" ] # and use the Kafka timestamp as your timestamp
    }
  }
  mutate { # Drop the data that we've already parsed or don't want
      remove_field => [ "[request][body][data][files][binary]", "[content][data][results][files][binary]", "message" ]
  }
}

output {
  if [@metadata][cloned] == "true" {
    elasticsearch {
      id => "dev-output-files"
      hosts => ["my-es-host1:9200", "my-es-host2:9200", "my-es-host3:9200"]
      index => "dev-files-%{+YYYY.MM.dd}"
      document_id => "%{[@metadata][id]}"
    }
  } else {
    elasticsearch {
      id => "dev-output"
      hosts => ["my-es-host1:9200", "my-es-host2:9200", "my-es-host3:9200"]
      index => "dev-%{[@metadata][kafka][topic]}-%{+YYYY.MM.dd}"
      document_id => "%{[@metadata][id]}"
    }
  }
}

[davewat]

I don't agree this is a bug. I believe you should have to include @timestamp in a whitelist if you want to keep it.

For example: If I am cloning an event to create a new event, I will then use prune/whitelist to keep the fields I want, which may not include @timestamp.

Prune should work across all fields equally, without arbitrarily assigning "@timestamp" a special status. As is, if I want to keep it, I whitelist it.