Topic 3: Explore tokenizing the recordedBy

[timrobertson100]

The current algorithm does not accommodate variation in recordedBy that includes multiple collectors.
For example, recordedBy will not be considered as overlapping between a record containing recordedBy=Tim Robertson|Nicky Nicolson and another with Tim Robertson.

@nickynicolson has previous work that attempts to parse recordedBy into tokens accommodating variety in delimiters used (, | etc). This is in Python, so not easily portable to Java.

To determine if it is worth exploring this approach, we could create a new table that tokenises the recordedBy String into an array of names, and then add a SQL JOIN to create a new occurrence table containing this field (e.g. a tokenizedRecordedBy). The clustering could be modified to use this field in both the blocking and the compare stages, and a report of the impact generated.

If this identifies useful links, the best approach to incorporate this into the clustering could be explored.

[dshorthouse]

recordedBy is a wild west. Here's another "tool", https://github.com/bionomia/dwc_agent. There's a tokenizing aspect to this (~ parsing) but there's also a structural scoring aspect to this (another basic method in that gem) that would help assert that "T. Robertson" is structurally similar to "Tim Robertson" but NOT "S. Robertson", given that you'd be able to find all such instances of agent strings with "Robertson". Likewise, that similarity score (arguably biases to Western names) would help differentiate "W.R. Smith" from "W.A. Smith" (poor) vs "W. Smith" from "W.A. Smith" (more acceptable) vs "William Andrew Smith" from "W.A. Smith" (good), https://github.com/bionomia/dwc_agent/blob/master/lib/dwc_agent/similarity.rb#L20

Regardless, perhaps a collaborative, generic spec test suite (and what should be our tokens) is a good goal. Maybe something like this: https://github.com/bionomia/dwc_agent/blob/master/spec/resources/test_data.txt

[dshorthouse]

If it's merely delimiters you're chasing for now, here's what's in that dwc_agent gem:

  SPLIT_BY = %r{
    [–|ǀ∣｜│&+\/;:]|
    \s+-\s+|
    \s+a\.\s+|
    \b(con|e|y|i|en|et|or|per|for)\s*\b|
    \b(?i:and|with)\s*\b|
    \b(?i:annotated(\s+by)?)\s*\b|
    \b(?i:coll\.)\s*\b|
    \b(?i:communicate?d(\s+to)?)\s*\b|
    \b(?i:conf\.?(\s+by)?|confirmed(\s+by)?)\s*\b|
    \b(?i:confirmada)(\s+por)?\s*\b|
    \b(?i:checked?(\s+by)?)\s*\b|
    \b(?i:det\.?(\s+by)?)\s*\b|
    \b(?i:dupl?\.?(\s+by)?|duplicate(\s+by)?)\s*\b|
    \b(?i:ex\.?(\s+by)?|examined(\s+by)?)\s*\b|
    \b(?i:in?dentified(\s+by)?)\s*\b|
    \b(?i:in\s+part(\s+by)?)\s*\b|
    \b(?i:och)\s*\b|
    \b(?i:prep\.?\s+(?i:by)?)\s*\b|
    \b(?i:purchased?)(\s+by)?\s*\b|
    \b(?i:redet\.?(\s+by?)?)\s*\b|
    \b(?i:reidentified(\s+by)?)\s*\b|
    \b(?i:stet)\s*\b|
    \b(?i:then(\s+by)?)\s+|
    \b(?i:veri?f?\.?\:?(\s+by)?|v(e|é)rifi(e|é)d?(\s+by)?)\s*\b|
    \b(?i:via|from)\s*\b|
    \b(?i:(donated)?\s*by)\s+
  }x

...cuz it's not just the usual suspects like "&", "|" that might be considered delimiters but language-specific delimiters and strings of text with meaning that may also be considered delimiters.

[dshorthouse]

And fwiw, the result of bi-monthly parsing for Bionomia produces a table with 180M rows for recordedBy of essentially two columns: agent string and gbifID. Would be happy to share if you want to give it a go.