Failure when using prune on a nested field

[3ygun]

Issue

Pruning nested fields within whitelist_names & blacklist_names doesn't work.

Need?

I have a large amount of fields (>100) which are being processed. I want to prune down to a subset of those fields. However, the data looks as follows with a nested field having the same name as a non-nested field:

{
  ...
  "blam": "blam message",
  "bar": "bar message",
  ...
  "foo": {
     ...
    "blam": "foo blam message",
    "bar": "foo bar message",
     ...
  },
  ...
}

If I want to keep only [foo][bar] and [blam] how do I reference them while removing the rest?

Test

I have the test: {"hi":"hello","test":{"bar":"1","ble":2}}

I want to run:

input {
  stdin {
    codec => json
  }
}

filter {
  prune {
    interpolate => true
    # Lets remove test.bar!
    blacklist_names => [ "[test][bar]" ]
  }
}

output {
  stdout { codec => rubydebug { metadata => true } }
}

But it fails to do anything:

I don't know if it's related to this issue: Update remaining plugins with Event API

[magnusbaeck]

Thanks for the clear report. You're probably right that it's the event API that's the problem. I'm currently away and won't be able to look at this until a day or two into next week though.

[3ygun]

@magnusbaeck Thanks!

[magnusbaeck]

Adapting the plugin to the new event API turned out to be harder than I thought.

The (white|black)list_names options support field name regexps, but what does that even mean for nested fields (that so far have been unsupported by this plugin)? Should we generate a list of field, [field], and [field][subfield] strings and match each item of (white|black)list_names against them? I feel there be many dragons here.

[boernd]

Why not just support white/blacklisting of top-level fields? Then it would work the same as before right?

[3ygun]

@boernd while that may work it doesn't solve the problem that, from a user perspective, the behaviour is inconsistent with the rest of logstash. I wasted at least an hour on the above as I expected it to work and it wasn't really documented otherwise.

@magnusbaeck notes below:

• As you have to add the interpolate => true flag to get the desired behaviour of %{field} behaviour such a list should only be computed when this flag is set. Furthermore, the interpolate documentations mentions it adds performance problems.
• However, after re-looking into the problem there might be deeper issues with getting even [field] to work. The [ ] in regex, I believe is a "character group", thus determining whether the user wants a "character group" or a logstash field name could become near impossible.
• There needs to be better consistency within the prune plugin or at the very least better documentation. However, the primary reason I used the plugin was for the (white|black)list_names which doesn't have replacement functionality in another filter plugin.
• I've also added clarification about the use-case above

[boernd]

I just realized this filter does not do anything in LS version 5.0.0 (not even filtering "normal"/non-nested fields. Adding this line seems to fix it but I'm not sure how this could work before (old api):

...
    event.overwrite(LogStash::Event.new(hash))

    filter_matched(event)
...