Coordinating provider datasets from SAMHSA (x2+) to NPI/NPPES and PALS

[jkdru]

Documenting Jan/Feb 2021 decisions and problems to solve for provider datasets.

For data recency & updates/archives:
-- Include two DATE fields per table: date_firstfind and date_lastfind
-- Our oldest date_firstfind is 2019-12-24, when @mariekers downloaded the 420 records initially provided by Code for Philly et al. in Feb/Mar 2020 data hackathon

For data privacy:
-- DB will upload only rows where archival_only = FALSE

[jkdru]

Audit tables will collect provrecs_ (provider records) as is.

1. Download CSV from SAMHSA's Buprenorphine Practitioner Locator (BPLoc)

Note these changes relative to 2020Q1 - 2020Q3:

The dataset includes several new columns. In roughly descending order of helpfulness:

SAMSHA colnames | MATchMapper colnames

• reachedPatientLimit | reached_patient_limit
• certifiedFor100 | xwaiver_100
• Find out: Does TRUE mean "at least 100" (including 275) or "exactly 100" (excluding 275)?
• latitude, longitude
  // Higher-res than same fields from SAMHSA's Find Treatment Locator (FTLoc)
  // To coordinate with lat, long for SITES (from FTLoc), round to nearest #.#### (e.g. 39.9482535 becomes 39.9483000; 39.9903107 becomes 39.9903000) -- see Issue Data flow for sites from directory sources and providers #20 re: sites data
• Check accuracy of geocodes for edge cases with CITY and/or ZIP errors (e.g. rows 527-529 "in Phila: ZIP error" cases)
• addressId | bprec_id // Appears to function as PK for individual records rather than individual locations

We can no longer download just the filtered view. So for 2021 (reopening Issue #16) we need to:

• Invoke "Get CSV of Full List", then apply location-based filters. TBD = which scope?
• 2020 focus on Philadelphia (jurisdiction) was stakeholder-specific.
• For 2021 focus on referral networks/treatment seekers, distance may be more useful. (E.g. Prevention Point mentioned in Feb 2020 orientation that individuals from NJ also seek help in Philadelphia.)

2. SAMHSA: xwaivers

• Address hiccup with apostrophe names (Issue Hiccup with apostrophe names #17)

3. NPI

• Search name of prescriber and the state to find NPI number (Issue Automate npi number collection based on names #2)

4. PALS
   ... (notes forthcoming)

[jkdru]

Sources for PROVIDER SPECIALTY filter:

• CMS reference https://taxonomy.nucc.org/
• Lists compiled from NPI and PALS data, Jan 2021

[jkdru]

In SAMHSA providers col certifiedfor100, TRUE does indeed mean "at least 100" (including 275). So rename our xwaiver_100 to xwaiver_100or275. Useful for stakeholders doing providers outreach: all FALSE values = xwaiver_30

In SAMHSA providers col reachedPatientLimit, only FALSE is helpful (7 of >500 names, 3 updated in 2020q4 or 2021q1); TRUE is inconclusive without further context/research