Rework tree parts of pipeline, using dated tree as source

[lentinj]

Replacement issue for #120, which is ultimately not the right approach.

Preliminary stages

Both stages stop worrying about only working with OTTs on the tree, and just work for all OTTs:

1. ID lookup resolution stage (Build table of OTT -> mappings, no trees involved):

• For each (filtered wikipedia dump) entry, get OTT ID, find associated OT ID tables (GBIF, NCBI, ...)
• If there's no overlap between the WD ids and OT ids, then bin row. We should have at least one corresponding ID.
• Merge sets of IDs. If WD has multiple IDs, take intersection with OT. If OT doesn't have corresponding entry, choose first (or whatever, point is we don't have multiple IDs beyond here)
• Output table of all IDs

2. Raw popularity resolution stage (OTT -> popularity):

• For each OTT in (1), dig out relevant page form wikipedia dumps, do stuff, assign popularity

New tree pipeline

1. Download trees that Jonthan needs using similar pipeline to current tree. tree_loading.load_metadata() will need to be fed same taxonomy file as we're using later in the pipeline. Consider having the OT & chronosynth as one download step in DVC, so we can just re-run both and hope they use the same tree.
2. Start with Jonathan's tree, run his pipeline bar the interpolation stage. Hope that polytomy nodes are already labelled as either virtual-nodes-to-be-ignored-in-polytomy-view polytomies (the current ones) or independently-reattached-nodes-that-remain-visible-but-have-a-grey-branch nodes (the new ones where nodes have been re-arranged). different types as above. Stripping birds/turtles isn't a problem, because they get replaced anyway.
3. Graft bespoke trees together
4. Polytomy resolution using existing random resolution. Mark nodes as (virtual-nodes-to-be-split-in-polytomy-view) polytomies
5. Resolve branch lengths to ages bottom-up. Remove (or not care about) branch lengths
6. Top-down conflict resolution in bespoke tree, delete entries that conflict with higher ages
7. (No chronosynth internal date-adding to bespoke tree. we either have NHX:age=x or branch length.)
8. Graft already OT subtrees onto our trees. Already polytomy-resolved / date pins from chronosynth applied.
9. Popularity calculations for entire tree (including any remaining subspecies from bespoke tree, Jonathan's will have them already removed) (stop caring about polytomy vs. popularity calculations, and just apply them post-resolution). Apply popularity based on OTT -> popularity map, percolate using existing rules (which preserves popularity from removed subspecies)
10. Remove subspecies
11. Top-down conflict resolution. If there's conflict with higher ages, remove ages until conflict goes away
12. Remove unary nodes (they are likely uninteresting, and make a mess of the tree rendering)
13. Re-interpoltate missing datesOnly store in DB only dates that are trusted, not interpolated.Interpolated only gets used for sliding_window, etc. (NB: biota is hard-coded as interpolated, even though it isn't).

[lentinj]

I think it's going to be easier to extract the ID/raw-pop code from CSV_base_table_creator as-is and simplify it's ID resolution logic at a later stage. Means I'm not rewriting the entire pipeline, and there'll be a simple table to compare when this stage is being rewritten.

However, the existing code tries to extract & match IDs against mrca nodes too:

tree-build/oz_tree_build/taxon_mapping_and_popularity/CSV_base_table_creator.py

Lines 189 to 207 in c3b31be

	m = mrca_ott_node.search(node.label)
	if m is not None:
	if m.group(3):
	logging.warning(
	f"Node has an @ sign at the end ({node.label}), meaning it "
	"has probably not been substituted by an OpenTree "
	"equivalent. You may want to provide an alternative subtree "
	"from this node downwards, as otherwise it will probably be "
	"deleted from the main tree."
	)
	node.label = m.group(1)
	# this is an 'mrca' node, so we want to save sources but *not* save
	# the ott number in node.data
	indexed_by_ott[m.group(2)] = node.data
	node.data["sources"] = {}
	elif node.is_leaf():
	logging.warning(
	f"Leaf without an OTT id: '{node.label}'. " "This will not be associated with any other data"
	)

Working purely against taxonomy.tsv, we won't have these available. But OTOH, this has to be dead code. mrca nodes don't have names (AFAICS) so the regex is wrong, and even if they did, we've thrown them away anyway.

None of the internal nodes seem to have sources here, so I'm just going to ignore this.

[lentinj]

@hyanwong @jrosindell

dated-complete-tree:remove_subspecies and dendropy_extras:prune_non_species are quite different. Respectively they:

• Based on OpenTree's taxonomy level, remove anything below "species"
• If there's no dedicated species node above the ones we removed, promote one to "species"

...and tree-build does:

• Remove nodes with bad phrases: "cf.", "aff.", "subsp.", "environmental sample", " cv.", " sp."
• Remove nodes without a space
• Special-case extinction nodes (unlabelled leaves with a length)

Which approach do we prefer? Note that most of the bad phrases still match things that are "species" in the OT taxonomy, so it removes things that dated-complete-tree wouldn't. But I'm not convinced it's "extinction nodes" are still a thing.

[lentinj]

Right, issue-131-new-date-pipeline has a ~entirely re-written tree pipeline. The major scripts are:

• download_opentree: Fetches what dated-complete-tree needs, not the simplified tree
• download_node_ages: Does the chronosynth-fetching
• date_tree: dated-complete-tree's main.py, with an output-tree-and-exit step wedged in. I've left this intentionally messy, so we can see what parts of the full pipeline we are using, and when we bow out
• taxon_map: The taxon mapping steps from CSV_base_table_creator, purely working against the taxonomy.tsv rather than applying to a tree straight away.
• tree_build: A monstrous step that combines most of get_open_trees_from_one_zoom, build_oz_tree, CSV_base_table_creator & make_js_files.
  • step_parse: Parse bespoke/orphan trees
  • step_graft: Inclusion syntax, grafting everything together into one tree
  • step_taxon: Assigns taxon map to the tree
  • step_popularity: Popularity assignment to tree, assuming raw_popularity has been done via. taxon_map
  • step_tidy: Some tree-tidying steps where dated-complete-tree didn't obviously do what we want.
  • step_treeprop: The precomputed geological / sliding-window / weighted-mean property calculations
  • step_jsnewick: Generating brief newick, cutmap directly from ete4.
  • step_output: Generate CSV table, array JS file outputs.
• versioned_outputs: Adds the version number to the filenames, and into the import.sql script.

Some notes & TODOs:

• Whilst (I think) it makes it to the end, and the individual parts are tested, I've not spent any significant time looking at the end result. It may not actually be that dated yet.
• There's no estimation of biota on the bespoke tree. We'd also have to plumb it into date_imputed, and stop imputed dates ending up in the CSV files
• We don't try to distinguish dated-complete-tree-polytomy-specials yet, and generally I suspect the polytomy handling between stages is a bit wonky.
• I've just noticed that exclude_taxa isn't wired up. It'd make more sense to add it as an option to taxon_map though, not tree_build.
• Similarly, SupplementaryTaxonomy.tsv isn't wired in to taxon_map
• csv_base_table_creator step hasn't been removed yet. Mostly to make it obvious that the above 2 haven't been done.
• The branch currently has a lot of dead code as I've been copying, not deleting. That'll have to be rooted out before merging.

Some help would be appreciated:

• Deciding what to do with non-species nodes, as above. I'm tempted to go with dated-complete-tree's approach, but the end result would be a bit different. Maybe we need some kind of hybrid?
• This repo currently bodges the missing sharks, I've not tried inserting the new tree.
• Generally having a read though date_tree & tree_build to make sure the pipeline sounds like what you're expecting.