The common algorithm for estimating race from names and geography is BISG (Bayesian Improved Surname Geocoding). As an example, suppose there is a customer Mary Johnson who lives in Essex County, NJ. If we want to predict her race only from her surname, we use nationwide Census probabilities such as P(Black ∣ Johnson) = 0.3441, P(White ∣ Johnson) = 0.5438, etc.
To improve the prediction, we add her county, using geographic weights such as the relative density of each race in Essex compared with nationwide. The resulting probability P(Black | Johnson & Essex) = (Black national prob × Black geographic weight) / ∑race (race national prob × race geographic weight) = 0.7393.
Finally, we add her first name. Under a Naive Bayes assumption that surname, location, and first name are conditionally independent given race, Bayes’ rule allows us to multiply the components: P(race ∣ surname & geo & first ) ≈ P(race) x P(surname ∣ race) x P(geo ∣ race) x P(first ∣ race). The probability P(Black ∣ Johnson & Mary & Essex) becomes 0.7158.
A summary of these probabilities is:
| Model Type | White | Black | Hispanic | Asian | Other |
|---|---|---|---|---|---|
| Surname Only | .5438 | .3441 | .0272 | .0074 | .0774 |
| Surname+Geo | .1766 | .7393 | .0244 | .0046 | .0551 |
| Surname+First+Geo | .2481 | .7158 | .0042 | .0013 | .0306 |
These calculations can be done with the R package wru (Who are you?). You will need a Census API key to download the Census data, available from https://api.census.gov/data/key_ssignup.html .
I was interested in measuring the accuracy of the BISG predictions. It is not easy to find real (not simulated) data linking name and race. However, several states make their voter registration records public. I requested and downloaded Florida voting registration data from Harvard Dataverse ( https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/UBIG3F ). I limited my study to a single Florida county, Lee, which had over 500,000 registered voters.
The Florida voting data required considerable data cleaning. Among the amusing data issues is that some Lee County voters registered with a specialty post office box zip code, or a zip code with a typo, or an out-of-state zip code. The first two were mapped to county level, and the third category was dropped.
There were 545,187 registered Lee County voters, distributed by race as follows:
| Asian | Black | Hispanic | Other | White | NA |
|---|---|---|---|---|---|
| 6912 | 32407 | 66677 | 9435 | 422333 | 7423 |
I performed the same BISG algorithm as above, against the Florida voter registration data for Lee County. The overall accuracy is 88%, but this is misleading because the Florida data is highly imbalanced toward the White race. A better accuracy measure is one that is calculated separately for each race, and I chose Balanced Accuracy = (Sensitivity + Specificity)/2. Sensitivity measures how well the algorithm finds people of that specific race, and specificity measures how well the algorithm avoids misclassifying people of other races into that specific race.
My Balanced Accuracy results of comparing BISG race predictions versus Florida self-identified race data is as follows:
| White | Black | Hispanic | Asian | Other |
|---|---|---|---|---|
| .8226 | .7179 | .9007 | .7251 | .5041 |
I think these results are pretty good for Hispanic and White, not as good for Black and Asian, and of course pretty poor for Other. Other includes Native American, Native Hawaiian, and others, and are a small overall percentage of the county population.
The citation for the Florida voter data is: Sood, Gaurav, 2017, "Florida Voter Registration Data (2017 and 2022)", https://doi.org/10.7910/DVN/UBIG3F, Harvard Dataverse, V2.
Here is my R code to predict Mary Johnson.
library(wru)
library(dplyr)
# 1. Create a 1-row dataset for Mary Johnson in Essex County, NJ (FIPS "013")
test_voter <- data.frame(
surname = "JOHNSON",
first = "MARY",
state = "NJ",
county = "013", # Essex County FIPS
stringsAsFactors = FALSE
)
# 2. Grab the live Census geographic data for New Jersey
nj_census <- get_census_data(
key = "xxxx",
states = "NJ",
age = FALSE,
sex = FALSE
)
# 3. Generate the 3 prediction variations
pred_baseline <- predict_race(voter.file = test_voter, surname.only = TRUE)
pred_geo <- predict_race(voter.file = test_voter, census.data = nj_census, census.geo = "county")
pred_first_geo <- predict_race(voter.file = test_voter, census.data = nj_census, census.geo = "county", names.to.use = "surname, first")
# 4. Format and clean output to perfectly fit the RStudio layout
combined_predictions <- bind_rows(
pred_baseline %>% mutate(model_type = "Surname Only"),
pred_geo %>% mutate(model_type = "Surname+Geo"),
pred_first_geo %>% mutate(model_type = "Surname+First+Geo")
) %>%
rename(
White = pred.whi,
Black = pred.bla,
Hispan = pred.his,
Asian = pred.asi,
Other = pred.oth
) %>%
mutate(across(c(White, Black, Hispan, Asian, Other), ~ round(., 4))) %>%
select(model_type, surname, first, White, Black, Hispan, Asian, Other)
# 5. Display the output summary
print(combined_predictions, row.names = FALSE)
# End of Mary Johnson prediction
##################################################################################
Here is my R code to measure the accuracy of the BISG predictions against Lee County voters. A considerable portion of the code is for data cleaning of the Florida voter registration data.
library(data.table)
library(dplyr)
library(stringr)
library(caret)
library(wru)
voter_file_path <- "xxxx/LEE_20220621.txt"
Sys.setenv(CENSUS_API_KEY = "xxxx")
raw_lines <- fread(
file = voter_file_path,
sep = NULL, # Read the row as a single string to lock layout shifting
header = FALSE,
col.names = "raw_row"
)
wru_ready_data <- raw_lines %>%
mutate(
# Extract structural components securely out of text blocks
surname = str_match(raw_row, "\\d{9}\\s+(\\w+)")[,2],
first = str_match(raw_row, "\\d{9}\\s+\\w+\\s+(\\w+)")[,2],
zipcode = str_extract(raw_row, "\\b3\\d{4}\\b"),
raw_race = str_match(raw_row, "\\b([MF])\\s+(\\d)\\b")[,3]
) %>%
# Filter out empty records or text extraction errors
filter(!is.na(surname) & !is.na(raw_race)) %>%
# Map Florida's state administrative codes to wru categories
mutate(
true_race = case_when(
raw_race == "5" ~ "white",
raw_race == "3" ~ "black",
raw_race == "4" ~ "hispanic",
raw_race == "2" ~ "asian",
raw_race %in% c("1", "6", "7") ~ "other",
TRUE ~ NA_character_ # Automatically standardizes missing responses to NA
)
)
# Clean data of invalid zip codes and reroute PO Boxes to residential ZCTAs
wru_ready_data <- wru_ready_data %>%
# Exclude out-of-state ZIP codes (Georgia 30xxx/31xxx, Alabama 35xxx/36xxx)
filter(!substr(zipcode, 1, 2) %in% c("30", "31", "35", "36")) %>%
# Map missing/typos and PO boxes directly to matching valid residential ZCTAs
mutate(
zipcode = case_when(
# Defunct / Typos / Missing data map to Central Lee County Baseline
zipcode %in% c("fl-NA", "33929", "33932", "33945", "33970") ~ "33901",
# PO Box explicit routing to residential census counterparts
zipcode == "33902" ~ "33901", # Fort Myers PO Box -> Fort Myers Residential
zipcode == "33906" ~ "33907", # Fort Myers PO Box -> South Fort Myers Residential
zipcode == "33910" ~ "33904", # Cape Coral PO Box -> Cape Coral Residential
zipcode == "33915" ~ "33919", # Fort Myers PO Box -> Cypress Lake Residential
zipcode == "33918" ~ "33903", # N. Fort Myers PO Box -> N. Fort Myers Residential
zipcode == "33994" ~ "33928", # Bonita Springs PO Box -> Estero/Bonita Residential
zipcode == "34106" ~ "34102", # Naples PO Box -> Naples Residential
zipcode == "34133" ~ "34135", # Bonita Springs PO Box -> Bonita Residential
zipcode == "34136" ~ "34135", # Bonita Springs PO Box -> Bonita Residential
TRUE ~ zipcode # Keep all other valid residential ZCTAs as they are
),
# Ensure wru recognizes the geographic county boundary for the fallback imputation
county = "12071" # FIPS code for Lee County, FL
)
nrow(wru_ready_data)
table(wru_ready_data$true_race, useNA = "always")
lee_county_test <- as.data.table(wru_ready_data) %>%
filter(!is.na(true_race)) %>%
mutate(
state = "fl",
surname = as.character(surname),
first = as.character(first),
zcta = as.character(zipcode) # Required column label mapping for ZCTA use
)
florida_zcta_layers <- wru::get_census_data(
key = Sys.getenv("CENSUS_API_KEY"),
state = "FL",
age = FALSE,
sex = FALSE,
census.geo = "zcta",
county.list = NULL # Required parameter boundary choice for independent ZCTAs
)
predicted_lee_county <- wru::predict_race(
voter.file = lee_county_test,
census.surname = TRUE,
surname.only = FALSE,
census.geo = "zcta",
census.data = florida_zcta_layers,
impute.missing = TRUE, # Forces wru to fall back to County priors if needed
skip_bad_geos = TRUE
)
# "53233 (10.1%) individuals' last names were not matched, but ZCTA's baseline geographic racial demographics to calculate the prediction.
evaluation_ready_data <- predicted_lee_county %>%
rowwise() %>%
mutate(
# Isolate which of the five columns held the highest posterior probability string
max_prob_col = c("white", "black", "hispanic", "asian", "other")[
which.max(c(pred.whi, pred.bla, pred.his, pred.asi, pred.oth))
]
) %>%
ungroup() %>%
# Convert fields into factor arrays to prevent tracking sequence errors
mutate(
true_race = factor(true_race, levels = c("white", "black", "hispanic", "asian", "other")),
predicted_map = factor(max_prob_col, levels = c("white", "black", "hispanic", "asian", "other"))
)
# Compute the final metric confusion matrix tracking loop
accuracy_report <- confusionMatrix(
data = evaluation_ready_data$predicted_map,
reference = evaluation_ready_data$true_race
)
print(accuracy_report$overall["Accuracy"]) # Overall model precision
print(accuracy_report$byClass[, "Balanced Accuracy"]) # Success broken down per group
End
No comments:
Post a Comment