Redwall Analytics

Amateur Urbanist Critique of Work-Live-Ride using 360k Fairfield County Parcels

Thu, 16 Jan 2025 00:00:00 +0000

MetroNorth New Haven Line Stations

Introduction

Connecticut has long had some of the highest income and home prices in the US. Although home prices have trailed national rate of appreciation in recent decades, from a high starting point, housing has gotten a lot more expensive in absolute terms and is still out of reach for too many. High housing costs have constrained population growth in many towns, likely slowed income growth by impeding business formation, and contributed to the State’s high costs and taxes. Starting in the 1920’s, many CT Towns started requiring large minimum parcel sizes in zoning, among other regulations, often with exclusionary intent, well documented in On The Line: How Schooling, Housing, and Civil Rights Shaped Hartford and its Suburbs. The effects of these policies have become acute due to the demand surge caused by the COVID-19 pandemic, which gobbled up residual excess supply left by the Global Financial Crisis. In 2021, a group called Desegregate CT Work-Live-Ride began making a series of reform proposals, a central one being Bill #6831 (WLR), to allow towns to opt into “Transit Oriented Development” allowing “by right” development of minimum densities and mixed use development near transit stations. This group consists of dozens of housing advocacy groups with backing from the Regional Plan Association (RPA), also non-profit whose largest contributors Board Members and often Executive leadership have often come from the New York real estate development industry.

One of the Desegregate group’s main points has been that only single family homes are allowed as of right on 90% of parcels in the State, while much less commonly for two family housing (and almost not at all more than that). Also, 80% of those single family parcels have been zoned 1+ acres, constraining housing supply growth increasingly over decades and driving up home ownership costs and sprawl. Finally, many towns have allowed only single family residential zoning around vital transit hubs. These points all are unfortunately true, but in my opinion, there is a bait and switch using the valid large lot issue as the reason we need the inflexible WLR, when densities near MNR have much lower percentages of residential housing and very few large nearby parcels.

The Connecticut Parcel and Cama data was first added to the Portal after laws requiring it passed in 2021, and offering the opportunity to look at the exact Polygon location data for almost 1.3 million parcels. This made me curious to investigate the lot sizes and housing density in those locations using the CT Geodata Portal and to explore what it all would mean for my Town of Greenwich. There are surely a lot of experts with more knowledge of these issues and stronger views on either side mine. I agree with most of the sentiments of the pro-housing reform coalition, but hope on the eve of the legislative session, this post injects what has seemed absent from the contentious discussions thus far. For full disclosure, I grew up in Stamford and live near a station in Greenwich (but not within a potential transit zone), and probably carry some of those biases against what might be done in a suburban location, but also have yet to see a multi-family residential project in my town which I opposed (likely putting me among the most pro-housing residents in my community). There is a lot of data cleaning and coding in this post, so please feel free to skip ahead to Results and Analysis to the tables and charts of findings and parting thoughts.

Collecting, Loading and Cleaning Data

I downloaded the Parcel File Geodatabase and 2024 CAMA property assessment data for each city. There are also separate files for county layers, but used the 2024 Basic Parcel Layers (including all counties) and filtered for Fairfield County and the last few Towns up the MNR line into New Haven County, the economic engine of the State. I get “unexpected geometry” warnings about interior rings, but discovered that changing to type = 3 while importing, converting the 3-dimensional XYZ Multi-Polygons to 2-dimensional XY Polygons for each parcel, which seems to work for my purposes. In the future, I would like to see if I can also look at the building footprints within the parcels, but this was enough for the purposes of this post. This was my first attempt at using GIS data on this scale, and so my knowledge is superficial, and suggestions are welcome.

Load and Clean FF Parcels

# Basic Parcel Geodatabase 
folder <- "~/Documents/Data/ct_state_data/"
file <- paste0(folder, "2024 Basic Parcel Layer.gdb")

# Reading and filtering for Fairfield County towns
ff_parcels <- read_sf(
  dsn = file, 
  query =
    "SELECT * FROM \"Basic_Parcels_2024\" where Town_Name in ('Greenwich', 'Stamford', 'Darien', 'New Canaan', 'Westport', 'Norwalk', 'Bridgeport', 'Shelton', 'West Haven', 'New Haven', 'East Haven', 'Trumbull', 'Easton', 'Redding', 'Bethel', 'Brookfield', 'Danbury', 'Newtown', 'New Fairfield', 'Ridgefield', 'Wilton', 'Weston', 'Stratford', 'Fairfield', 'Monroe', 'Ridgefield', 'Orange', 'Milford', 'Derby')", 
  type = 3)

# Clean parcels
ff_parcels <- ff_parcels[sf::st_is_valid(ff_parcels),]

# Fix Danbury links
ff_parcels[ff_parcels$Town_Name == "Danbury", ]$CAMA_Link <- 
  paste0("18500-", ff_parcels[ff_parcels$Town_Name =="Danbury", ]$Parcel_ID)

There were several thousand parcels which were invalid, and the only way I could manipulate the data with the {sf} package, was by removing them with sf::st_is_valid(). Also, Danbury’s disclosure was missing all property links, but I was fortunately able to extract and parse from other fields. The polygon shapes can be seen inside the NAD83 Connecticut boundaries. I spent some time trying to convert this to a lon-lat Coordinate Reference System (CRS), but in the end, discovered could do everything I needed with the original CRS distances. Below is the Shape of the parcels, which are I believe are called State Plane Coordinates.

# Towns along the MRN property Polygons
print(ff_parcels$Shape)

## Geometry set for 359565 features  (with 10 geometries empty)
## Geometry type: POLYGON
## Dimension:     XY
## Bounding box:  xmin: 730512.2 ymin: 554931.1 xmax: 1002166 ymax: 756258.3
## Projected CRS: NAD83 / Connecticut (ftUS)
## First 5 geometries:

## POLYGON ((868586.9 613438.8, 868509.1 613356.3,...

## POLYGON ((868361.3 613493.1, 868247.5 613380, 8...

## POLYGON ((868662.6 613522.2, 868586.9 613438.8,...

## POLYGON ((867755.4 613622.7, 867744.1 613443.5,...

## POLYGON ((867505.7 613668.3, 867622.2 613647, 8...

The Geodatabase unfortunately does not currently have the property assessment data, including the “state use” codes (ie: Commercial, Residential (Single Family, Multi-family), Industrial), which are the only way I could sort residential properties, and within those, if more than one family would be allowed in a location. The problem with the CAMA data is that was not contributed by Towns with consistent standards (as I have often found with data submitted by 169 often small towns probably lacking data infrastructure). While the data could be a lot cleaner and more uniform, my first recommendation would be to clean up the state use codes, and include them in this GIS database, which would streamline things for those working with the raw data instead of the online Parcel Viewer.

Load and Clean CAMA Data

file <- paste0(folder, "/2024_Connecticut_Parcel_and_CAMA_Data_20250111.csv")
cama_data <- data.table::fread(file)
cama_data[, link := re2::re2_replace_all(link, " ", "")]
cama_data <- janitor::clean_names(cama_data)

# Fix some Darien links not matching ff_parcels data
cama_data[
    property_city == "Darien"
    , link := data.table::fifelse(
      re2::re2_detect(link, ".*\\-\\d$"), 
      re2::re2_extract_replace(link, "^(.*)\\-(\\d)$", "\\1-0\\2"),
      link
    )]

# Fix missing Bridgeport links
cama_data[
  property_city=="Bridgeport", 
  cama_site_link := paste("0", cama_site_link)]

# Clean up any whitepace to make sure of joins
cama_data[, cama_site_link := trimws(cama_site_link)]

The full extent of the work in progress nature of this data can be seen in the {skimR} summary below. Some places where the state might clean up this dataset are: eight towns haven’t populated the town_id field, and various other fields are incomplete. There are two zoning fields (zone and zone description), but no uniform structure to these. An academic from Connecticut was Founder of the National Zoning Atlas, an impressive data-oriented project, which led to the initial recognition of the extent of the large lot zoning problem in the State. It doesn’t appear that this data can be downloaded and combined with other data, but tying the parcel data to to minimum zoned lot sizes would be big opportunity. The Zoning Atlas data also doesn’t include Coastal Overlay or other flood plane data, which seems like a significant factor to zoning for many parcels along the MNR line (if you look at the train map above).

For now, the most important field for the purposes of this exercise is state_use which is never NA, but in four towns had an empty string for all properties and overall has over 1600 unique categories just in this subset of Towns. state_use for a single family in most towns is “101”, but there are many others coded as simply “100” or starting with “1” followed by other formats. If I had more time, I might be able to do a better job predicting state_use with state_use_description, but leave that for another time. For this reason, all data summaries below had to be looked at as approximate and by no means the final word on density or mix of land use near stations.

In order to get train station coordinates, I found the USDOT Intermodal Passenger Connectivity Dataset (IPCD) with 15,000 transportation hubs around the US. This offers the option to conduct future analyses around other transport hubs, like bus routes. It took a few tries to get the MNR lines, for example, Cos Cob is mistakenly listed in NY state, and some locations came up more than once if there was both an Amtrak, MNR or bus stop at that location. I generally prefer data.table, but this doesn’t work for data manipulation with sf objects. I was able to get away with it here using only the X, Y coordinates, and then convert back to sf further downstream.

Load Train Station coordinates

file <- 
  paste0(folder, 
  "NTAD_Intermodal_Passenger_Connectivity_Database/Intermodal_Passenger_Connectivity_Database_(IPCD).shp"
  )
trains <- st_read(
  dsn = file, 
  query = "select * from \"Intermodal_Passenger_Connectivity_Database_(IPCD)\" 
    where METRO_AREA LIKE 'Bridgeport-Stamford-Norwalk CT' AND MODE_RAIL = 1")

## Reading query `select * from "Intermodal_Passenger_Connectivity_Database_(IPCD)" 
##     where METRO_AREA LIKE 'Bridgeport-Stamford-Norwalk CT' AND MODE_RAIL = 1'
## from data source `/Users/davidlucey/Documents/Data/ct_state_data/NTAD_Intermodal_Passenger_Connectivity_Database/Intermodal_Passenger_Connectivity_Database_(IPCD).shp' 
##   using driver `ESRI Shapefile'
## Simple feature collection with 31 features and 52 fields
## Geometry type: POINT
## Dimension:     XY
## Bounding box:  xmin: -73.62515 ymin: 41.02125 xmax: -73.13083 ymax: 41.3981
## Geodetic CRS:  WGS 84

data.table::setDT(trains)

# Filter unique stations by point_id and select columns
trains <- unique(trains, by="POINT_ID")
trains <- 
  trains[, .(X, Y, POINT_ID, ADDRESS, METRO_AREA, FAC_NAME, CITY, STATE, ZIPCODE)]

DT::datatable(trains)

Code for Filtering and Aggregating Parcels within Ranges

This section has the code to load and find lots within progressively expanding radiuses for each station. My custom function, get_parcels_in_geo_range(), loads data from an IPCD point and filter CAMA points within a specified parameter distance. Again, I’m relatively new to GIS data, so it took me a long time to figure out how to create the station {sf} data.frame in a long-lat point XY dimension and with a CRS which could properly filter the ff_parcels for distance with st::st_is_within_distance(). First I had to load as CRS 4326, then transform to 2234 for distance calculations in accordance with my ff_parcels sf formatting. This took a long time to figure out, and I still don’t completely understand how it is working.

Get Parcels in GEO Range

get_parcels_in_geo_range <- function(point_id, dist_range) {
  
  # Using trains, cama_data, and ff_parcels from the global env
  
  # Get train station
  train_station <- trains[trains$POINT_ID == point_id]
  train_station <- data.frame(lon=train_station$X, lat=train_station$Y)
  
  # Convert to sf
  station <- 
    sf::st_as_sf(train_station, coords = c("lon", "lat"), crs = 4326) %>% 
    st_transform(crs = 2234) 
  
  # Find lots with components within dist_range
  wd <- 
    st_is_within_distance(ff_parcels, station, dist = units::set_units(dist_range, "mile"))

  # Drop any lots where none of the components are within dist_range
  parcels <- ff_parcels %>% filter(lengths(wd) > 0)
  
  # Convert to data.table
  data.table::setDT(parcels)
  
  # Clean names
  parcels <- janitor::clean_names(parcels)
  
  # Keep lots >=0
  parcels <- parcels[shape_area >= 0]
  
  # Copy cama_data to leave in place
  cama_data <- copy(cama_data)
  
  # Clean up any extra spaces in Cama link
  parcels[, cama_link := re2::re2_replace_all(cama_link, " ", "")]
  
  # Join data on "link" = "cama_link"
  parcels <- cama_data[parcels, on = c("link" = "cama_link")]
  
  # Return
  return(parcels)
}

In prepare_town_data_frame(), I had to use trial and error to figure out towns where the standard codes were not picking up residential properties. There were varying codes for single family home, condominium or apartment building, so I did my best to capture them all as shown in the resi_codes vector below. Unfortunately, I had to leave the job of separating single- and multi-family properties for a later time. I went with number of dwellings, which doesn’t distinguish between single and multi-family, but does give an overall sense of density in those zones.

Prepare Town data.frame

prepare_town_data_frame <- function(towns) {
  
  # Remove towns outside FF county without parcels
  towns <- towns[sapply(towns, nrow) > 20]
  
  # Convert list of towns to a single data.table
  towns_df <- rbindlist(towns, fill = TRUE, use.names = TRUE, idcol = "station_name")
  
  # Add an indicator and filter keeping properties thought to be residential derived by trial and error
  resi_codes <- 
    c(800, 801, 802, 803, 805, 899, 100, 101, 102, 103, 122, 104, 105, 108, 109, 172, 1010, 1040, 1110, 1012, 1015, 1050, 1111)
  towns_df[, resi := state_use %in% as.character(resi_codes)]
  towns_df[, resi := fifelse((resi == FALSE & state_use_description == "Residential"), TRUE, resi)]
  towns_df[, resi := fifelse((resi == FALSE & state_use_description == "Commericial"), FALSE, resi)]
  towns_df[is.na(resi), resi := FALSE]
  
  # Drop empty links
  towns_df <- 
    towns_df[!link %in% c("48620-", "77200-", "57600-", "68170-", "86370-", "52980-") ]

  return(towns_df)
}

prepare_station_summary() summarizes all towns based on properties which were not identified, total residential properties, mean acres of residential dwellings, total number of properties above 1/2 acre and 1 acre, and the total acres of residential land within the specified distance. I also calculated the percentage of residential land within the total area in the half mile radius, which for most stations was more than half.

Prepare Station Summary

prepare_station_summary <- function(towns_df) {
  
  # Find unmatched links
  missing <- towns_df[is.na(pid), .(unmatched = .N), station_name]
  
  # Count total resi properties near specified station_name
  total_resi <- towns_df[resi == TRUE, .(total_dwellings = .N), by = station_name]
  
  # Calc mean acres and counts of 1/2 and 1 acre parcels
  ff_summary <- dplyr::distinct(towns_df[resi == TRUE], shape, .keep_all = TRUE)[
    , .(
      avg_acres_dwelling = round(mean(shape_area)/43560, 2), 
      `total_half_ac+` = sum(shape_area>0.5*43560), 
      `total_1_ac+` = sum(shape_area >43560))
    ,  station_name]
  
  # Count total acres and resi acres near specified station_name and dist_range
  resi_land <- dplyr::distinct(towns_df[resi==TRUE], shape, .keep_all = TRUE)[
    , .(resi_ac = sum(shape_area)/43560), station_name]
  total_land <- dplyr::distinct(towns_df, shape, .keep_all = TRUE)[
    , .(total_ac = round(sum(shape_area)/43560, 1)), station_name]
  
  # Aggregate components into final table by station_name
  ff_summary <- merge(ff_summary, total_land, by = "station_name")
  ff_summary <- merge(ff_summary, resi_land, by = "station_name")
  ff_summary <- merge(ff_summary, total_resi, by = "station_name")
  ff_summary <- merge(ff_summary, missing, by = "station_name")
  
  # Round digits
  ff_summary <- ff_summary[
    , pct_resi := round(resi_ac/total_ac, 1)][
      , resi_ac := NULL]
  
  # Return
  return(ff_summary)
}

I wrote get_stations_summary() to iterate over all the train stations at a given dist_range in the select trains data.frame.

Get Stations Summary

Results and Analysis

Below is a summary table of stations within 1/2 mile of stations with several showing more or less than the 500 acres (expected in 1/2 mile radius). This is probably because some stations may be close to the water or include significant portions that are highway, so has a smaller area of land which may be developed for any purpose. Others may have a large property (ie: cemeteries, golf clubs, parks and schools) with an address within, but extending outside the radius. Bethel’s numbers look off, but it is missing most “state use” codes, so it is difficult to accurately account for what parcel usage. For a sense of the overall accuracy for a particular station, the “Unmatched Properties” column shows how many of the parcels were not classified by a “state use” code and were left out of the calculations. For example, I still have a lot of properties near Darien’s 33 West Ave station which are not classified. The proposed legislation has specified 15 to 30 units per acre minimum density as of right within these zones, so demonstrating how significant the change for adopting WLR could be for residents of those neighborhoods. Overall, there are no stations with average residential lots greater than 1 acre at any of the 26 stations.

(#fig:0.5_miles)Summary of Properties within 1/2 mile of stations

Looking at density per acre for the portion of the zone including residential state_use codes, seven out of 26 stations have averages of more than 1 acre per dwelling in the first mile with more after 1.5 miles, much smaller numbers than the 80% average in the state, but still a lot. All of the lower density stations were much further from Grand Central. I don’t know the circumstances at those stations, but it does support the claim that some towns may have allowed sparse density at these valuable locations. Two stations have average lot size well above 1 acre in the first 0.5 miles, but then drop below 1 acre further out. Please hover over the lines to more better discern the station and the density at a given point, or select labels on the right to drill down on any location.

(#fig:average_acres)Average acres per dwelling is low around many stations

The next chart shows the number of 1+ acre lots by mile from stations, showing there are few within 1/2 mile at stations closest to NYC in Greenwich, Stamford and Norwalk for example, but there are lower densities much further up the line. While there are approximately 1,000 1+ acre lots near the 26 MNR stations, there are 10x that at 1.5 miles, and 25x 3 miles further away. Once you get a few miles away from stations, there is a lot more land in some locations, so this is my point that the remedy seems to miss the opportunity to address the concern.

(#fig:average_acre_lots)There are not many large parcels near trains at most stations

Perspective on Greenwich Topology and `WLR` Impact

I have done my best to read the revisions of WLR Bills over the last few years, it really hard to understand the specifics of what my Town would be signing up for, and I’m not sure I would rely on the framing from either side of the debate. Along with five other towns, Greenwich has a population just above the 60,000 cutoff, above which the minimum “by right” density requirement jumps from 15 to 30 dwellings per acre. It is unclear if areas around all four stations would need to adopt those minimum densities or just a few to qualify, or if there are other formats which could satisfy the requirements. Given that this has been discussed for four years and so little has been clarified (at least to me), I can only assume it has been intentionally left vague or so complicated that nobody really knows. In the maximum four station scenario, and if building at that scale given the environmental constraints (discussed below), it could double the number of Town-wide dwellings (if half of the space around each station could be built upon), a substantial change.

From the perspective of a Greenwich resident, it is an unfortunate that the train line runs through the Southern-most tenth of Town, and that larger lot zoning spreads out to the Merritt Parkway and beyond (3-10 miles away). Adding density closer to the Post Road (just 1 mile from the MNR and 2 miles from the Sound), instead of modestly-zoned neighborhoods near the stations in the “Coastal Overlay Zone” (COZ), seems like a possible compromise. Transit zones centered around the 311 Bus stops along the Post Road, which in most cases would be closer to basic needs shopping like grocery stores and a lot more large residential parcels. It is also further from the fragile COZ. If my estimates are correct, there are currently less than 100 1+ acre residential lots within 1/2 mile of four stations (almost all of those in the COZ). Within one mile North of stations, there are about 400 greater than one acre lots, and 1,200 within two miles. Absent the WLR discussion, this seems like a more logical place to add density. Coupled with improved bus service and high quality bike lanes to trains stations, concentrating more density around the Post Road closer to the geometric center where there are larger plots, but still close to the trains, seems more natural than the requirements of WLR.

On the map below, significant portions of the transit zones (roughly estimated in the red circles below) around all of the train stations are in the COZ (shown in blue lines). The red arrows show possible locations about one mile North of stations closer to the Post Road (faint white along the top of the map). As it is written, WLR would be remaking neighborhoods composed entirely of smaller lots (the kind Desegregate supporters often say they believe should be legal everywhere), allowing developers to construct 10 unit buildings in between 1/4 acre single family parcels without any input from current owners is going to seem punitive. Bundling small parcels together into viable lots seems likely to be a slow and contentious process. A menu of options, like allowing the zone to be shifted North to the Post Road with bike lanes, combined possibly with “missing middle” changes near the MNR stations, allowing duplex and triplex (where they are not now) possibly with a maximum height, easy approval for ADU’s and reduced parking minimums. I still don’t understand why the discussion never includes MNR parking lots, which have inefficient parking policies and much more sparsely used since the pandemic. If the Town is going to grow meaningfully, I can’t see how it does that without a bridge North into actual large lots.

Greenwich Metro North Station Half Mile Radius and One Mile Pointers

Final Thoughts

There seem to be many options, but the same ill-fitting one keeps coming back in each round. I admit that there is strong resistance to building anywhere in Greenwich, and along the Post Road would be no exception, but if the goal is to get momentum behind more housing, which I think is overdue, community buy-in seems important. I’m skeptical that my town could sign on for WLR in its current drafting given the points above, but hope this will be the round where the two sides start compromising. If there are points I’ve got wrong about how WLR would work or otherwise, I’m just an observer who reads a lot and looks at numbers, and appreciate being educated and would be happy to edit my blog post.

NYED Data Explorer Shows 15 Years of Charter School Success

Wed, 22 Feb 2023 00:00:00 +0000

NYED Data Explorer filtered for “All Students” ELA Aggregated Annual Test Scores

Introduction

Three years ago, in the course of building personal projects in R using public data from Connecticut, I wrote How Does Stamford Charter School for Excellence do it?. “Stamford Excellence” was achieving remarkable results in a district with proficiency below 50%. Then and now, the school receives little recognition, and their effort to open a second Connecticut location in Norwalk, has been stymied as of this writing. My interest in these exceptional schools, and possibly the blog post, led to a new position as Director of Data & Research at another high-performing Bronx charter school network, South Bronx Classical Charter Schools. Naturally, when I discovered 15 years of NYED assessment data, the interest to clean and free this data for others to discover in a Shiny app, was immediate. The opportunity to also feature Classical’s stand-out performance didn’t hurt my motivation, although this post and the app were built in my spare time, and do not represent the opinions of South Bronx Classical Charter Schools. Unlike many past Redwall posts, this one will not have code, and will be primarily to explain the data and show how to use the app.

Classical Charter Schools and the Charter Debate

Classical Charter Schools operates four schools demonstrating outstanding academic performance in an under-served community, where there is not much evidence of educational accountability (based on the results which will be shown below). With approximately 90% economically-disadvantaged students, it achieves close to 90% proficiency in both ELA and Math, on par with the wealthiest NYS districts, while spending much less per pupil than DOE schools. This is accomplished with a variety of strategies, but most importantly for me, a big commitment to collecting and using data to make decisions. This post will show the result of extensive data cleaning, of over 3 million rows of school performance data, from the NYED Data site. Despite five year’s of daily experience manipulating data with code, untangling these releases took much more effort than I care to admit. After the recent broad drop in proficiency, related to the COVID, and the renewal of public debate about lifting the cap on new charter schools, it seems like there has never been a better time to make clean and accessible school performance data available. This post launches a “minimum viable” version of a Shiny App called NYED Data Explorer (accessible in this blog post below), which I hope may put to rest any questions over whether more charters should be allowed.

Thoughts on NYED and Open Public Data

As with every public data site I have worked with, the NYED’s disclosure strategy sometimes feels like it intends to make it hard to access a clean longitudinal data set. Disclosures occur annually, and are often in the inconvenient form of a MS Access Database, while other years shift to csv, xlsx or tab separated formats. Sometimes the Access databases include tables for enrollment, but leave them out in other years. In some years, only the current year is disclosed, and others also include the prior year. I was able to extract and separate these painfully, year-by-year from the command line using mdbtools. When the number of students in a group is below a threshold, fields are suppressed, which is common, but the NYED data denotes these cases with varying notations in different years. Sometimes, assessment data is disclosed in tables by grade, and others, all grades are stacked together in one table. Disclosed subgroups are usually male/female and ethnicity, but several years include ethnic group by gender. Important fields are added, and then disappear, such as in 2022, dropping “Mean Scale Score”, which showed the average score of a cohort at a school in most of the past years. As a result, I will concentrate on the “Pass Rate” for now, which is the number of students scoring in Level 3 or 4 divided by total test takers. The cutoffs for levels change every year as does the test difficulty, so “pass rates” are also not an objective measure over time.

Data Collection Considerations

Schools record and report student attributes, but in many cases, it seems likely that these have differences in interpretation, inaccuracies in collection and timing differences. Tracking race and ethnicity accurately within a single institution with changing personal is likely to be complicated, much less across 4,000+ schools, all with varying, often manual processes. There may be dozens of data filings a school has to make over the calendar year to local DOE as well as the State systems. Even if a scholar is classified correctly, there will surely be many cases where the totals may not be added up or transposed correctly into reporting systems. Enrollment data changes through the year, but is primarily recorded at “BEDS” Day. There are many cases, especially before 2012, where the number of test takers at the exams in the Spring exceeds the total enrollment in a grade reported at the same school. In fact, in the early years of this data, the aggregate number of test takers often exceeded the total enrollment in all schools in a subgroup. This may be still an error in my data cleaning, but is unclear as of this writing. The intention of this app is to give the most accurate representation possible. Please understand that this is impossible given the nature of the data, although it lot better than anything else I have discovered up until now.

Navigating the App

The interactive {golem} Shiny app, which will be shown below (please be patient if it takes a moment to load), was built with the {shinyWidgets} package selectizeGroup module, allowing “conditional drop-down filtering”. By reducing the subset of each filter with the other filters, it is possible to narrow down to desired groups with little or no latency. I discovered this by accident after trying a lot of things which didn’t work nearly as well, and still feel that it may not be well known as it deserves to be (see discussion in Introducing the Redwall IRS SOI Tax Dashboard) for more detail.

title=“Embedded NYED Data Explorer App” allowfullscreen

In the title image at the very beginning of this post, the app is shown filtered for the Subgroup and Test of “All Students” and the “ELA”, respectively. Using the ellipsis on the top left of the interactive app, try those filters, because when loaded, the app shows all subgroups and tests taken over the last 15 years (counting the same students in multiple groups). Once filtered, the population students taking the ELA (including both DOE and charter public schools), should decline from 1.2 million in 2007 to 900,000 last year. When using the app, it is important to make sure that the subgroups selected are mutually exclusive. For example, a “Black or African American” student may also be included in “Students with Disabilities”, so choosing both of those subgroups would double count the total tested and give invalid pass rates.

Again using our “conditional drop-down filtering”, the “Needs/Resource Category” labels are NY State classifications, which is helpful because it includes groups for NYC (DOE schools) and for all NYS Charters. My interest is largely Charter schools around NYC, so choosing “NYC” in the “Needs/Resource Category” reduces the County selections offered to only the desired five (“QUEENS”, “BRONX”, “KINGS”, “NEW YORK” and “RICHMOND”). Once NYC counties and the Needs/Resource Group are selected, the “Lift” column populates. “Lift” is defined as the aggregated pass rate of the selected group, divided by that of all the other test takers at schools in those same geographies and subgroups (not including the selected schools). So in this case, the “Lift” indicates that 308,000 students in NYC DOE schools have a pass rate, which in 2022, is 91.3% of the other schools not in the subset (in this case charter schools)

NYED Data Explorer filtered for All Students in NYC

After selecting those counties, changing to “Needs/Resource Category” to “Charters” now filters for only charters in those counties (essentially all NYC Charters), which flips the “Lift” for 2022 to the inverse of the previous example. The number of Charter students taking the ELA test has risen from ~6k in 2007 to ~66k in 2022 as new schools have opened and more students at those schools reached the upper grades. The “Lift” for Math at charters relative to the baseline DOE schools is higher than for ELA at 18%.

NYED Data Explorer filtered for All Students at NYC Charters

The demographics of the charter schools are also much different than the overall NYC population of students, and this matters, as we will now show. We can switch our single subgroup from “All Students” to the two largest subgroups at most charters, “Black or African American” and “Hispanic or Latino” (shown below). With these filters, the “Lift” for NYC charter’s ELA pass rate rises to 40%, because attending charter schools has a much bigger effect for these two subgroups. We haven’t shown, but leave it for the user to switch Test to “Math” to see the impressive 90% “Lift” for Math for charter over DOE students.

NYED Data Explorer filtered for African American and Latino Students at NYC Charters

At Classical like many NYC charters, about 90% of students also fall into these two subgroups, so it makes sense to see the Lift comparing the most relevant cohort of students. Though we have filtered down to charters in NYC bouroughs, there are still a lot of schools to scroll through to further subset, so starting to type the name of the desired schools can speed up selection. Once the Classical schools are selected, we can see that almost all South Bronx Classical Charter students are tested, and just under 90% are “proficient” in Math, and there was very little “learning loss” due to Covid (as there has been widely across most other schools in the United States). “Lift” shows that students at South Bronx Classical are proficient in Math at about 3.7x the rate as those in the DOE system. Pass rates for ELA are almost the same in absolute terms, but “Lift” is lower than for Math at 238%, because the DOE pass rates are higher in that subject.

NYED Data Explorer filtered for African American and Latino students at South Bronx Classical

Although Classical Charter Schools has been especially successful in navigating the Covid disruption, it is not the only network with persistent strong performance. Please also consider also looking at Success, Icahn, Bronx Excellence and Zeta, which of which operate multiples schools demonstrating excellent results over time, and there are surely others.

NYED Data Explorer filtered for Students with Disabilities at South Bronx Classical

Although entry to NY public charter schools is by lottery, critics have sometimes charged that charters avoid difficult cases. Changing the subgroup to “Students with Disabilities”, the 106 Students taking the Math test at Classical were about 7x more likely to be proficient. This is more a comment on the very small number of passing SWD students at district schools, but is still striking to see what is possible. Students with Disabilities at Classical did better in absolute terms than the general population at most schools in the State. Another interesting subgroup, which is virtually gone from Classical in most years, are English Language Learners. Although 30% of entering students are ELL, with such high pass rates, most students pass the ELA in 3rd or 4th grade, and “test out” of ELL status.

Future additions

Making the data cleaning steps reproducible is one goal, but given the number of manual steps to collect the raw data out of MS Access databases, this is not easy. Now that the data is mostly clean, and the app is up and running, I will be adding further tables and graphics to aid in explorations to compare the trajectory of individual schools. Approximately 80 charters have come and gone for poor performance. It would be interesting to see how many failing DOE schools have also shut down, and the relationship between that and performance. We have narrowed our filters down to Counties, but zip codes might be even more revealing and not difficult to add. I would like to add the ability to download subsets of the data and also to extract png images of future graphics.

Conclusion

The goal of the NYED Data Dashboard, as with many of my past “Redwall Analytics” projects, was to free valuable data from an inaccessible open repository, especially where the honesty surrounding the debate might be improved. My walk through is opinionated, but I hope is supported by data, which is now available for all to test and reproduce. Although I focused on NYC schools, the data includes all NYS public schools. The comments section of this blog is open, and I would welcome feedback on how to improve the app, different interpretations of the data or how to boost awareness of it.

Handling larger than memory data with {arrow} and {duckdb}

Sun, 27 Nov 2022 00:00:00 +0000

Setup

library(data.table)
library(glue)
library(arrow)

## 
## Attaching package: 'arrow'

## The following object is masked from 'package:utils':
## 
##     timestamp

library(duckdb)

## Loading required package: DBI

library(tictoc)

## 
## Attaching package: 'tictoc'

## The following object is masked from 'package:data.table':
## 
##     shift

library(ggplot2)
library(scales)
library(dplyr)

## 
## Attaching package: 'dplyr'

## The following objects are masked from 'package:data.table':
## 
##     between, first, last

## The following objects are masked from 'package:stats':
## 
##     filter, lag

## The following objects are masked from 'package:base':
## 
##     intersect, setdiff, setequal, union

library(bit64)

## Loading required package: bit

## 
## Attaching package: 'bit'

## The following object is masked from 'package:data.table':
## 
##     setattr

## The following object is masked from 'package:base':
## 
##     xor

## Attaching package bit64

## package:bit64 (c) 2011-2017 Jens Oehlschlaegel

## creators: integer64 runif64 seq :

## coercion: as.integer64 as.vector as.logical as.integer as.double as.character as.bitstring

## logical operator: ! & | xor != == < <= >= >

## arithmetic operator: + - * / %/% %% ^

## math: sign abs sqrt log log2 log10

## math: floor ceiling trunc round

## querying: is.integer64 is.vector [is.atomic} [length] format print str

## values: is.na is.nan is.finite is.infinite

## aggregation: any all min max range sum prod

## cumulation: diff cummin cummax cumsum cumprod

## access: length<- [ [<- [[ [[<-

## combine: c rep cbind rbind as.data.frame

## WARNING don't use as subscripts

## WARNING semantics differ from integer

## for more help type ?bit64

## 
## Attaching package: 'bit64'

## The following object is masked from 'package:utils':
## 
##     hashtab

## The following objects are masked from 'package:base':
## 
##     :, %in%, is.double, match, order, rank

library(microbenchmark)
uscompany <- "~/Documents/Projects/uscompanies/data/"
options(scipen = 999)
knitr::opts_chunk$set(echo = TRUE, warning = FALSE)

Introduction

It has been a while since loading Large, Messy CSV using {data.table} fread with CLI tools, but there are fast developing tools, which we didn’t fully understand at the time, making the problem discussed more manageable. When we left off, we had used scrubcsv split our problematic csv into two parts, a clean 30 million-row, 28-column data set of US business addresses, and a separate 1 million-rows with 22 or fewer columns. It was interesting to see what could be done on the CLI, but stacking the two significantly different subsets seemed cumbersome, considering varying column names and types, when one would not fit in memory. The objective of this posting will be to explore how to load the two pieces of our address data into arrow tables, standardize variable types, stack and queries. One of the main themes of this blog all along has been, how to become as agnostic as possible to data size, it feels like this moment may have arrived with {arrow} and {duckdb}.

Other Resources

We usually try to give credit to commentators who have made our learning possible, so here are a few that have made this post possible:

Loading an `{arrow}` Table

First, we load the larger 30 million row clean data set, which has 28 columns with {arrow}’s read_csv_arrow(). In the previous post, we were using a 2015 iMac with 8G of RAM, and now, our new MacBook Air m1 with 16G of RAM. When we ran the same function on the older iMac taking the data instead as data.frame, the same data took almost 10 minutes to load, but on the new machine, it took about 1 minute (so the time savings are about 90% from the m1). We also tried to load on the m1 as a data.table with fread(), but gave up after about 15 minutes. Here, we will actually choose the option to take the input as an {arrow} table (setting as_data_frame = FALSE), which reduces load time a little further to 45 seconds. As an {arrow} table, the 7G of data takes up only 283kB of memory to be accessed via {dplyr} in RStudio (instead of 725MB if loaded as a data.frame). In the context of this size of data and machine, {data.table} doesn’t seem like a viable solution compared to {arrow}, either as a data.frame or an {arrow} table, much as we love it.

tic()
cleaned_arrow <- 
  arrow::read_csv_arrow(paste0(uscompany, "scrubbed.csv"), as_data_frame = FALSE)
cleaned_arrow

## Table
## 30266526 rows x 28 columns
## $COMPANY_NAME <string>
## $SIC_CODE <int64>
## $SIC_DESCRIPTION <string>
## $ADDRESS <string>
## $CITY <string>
## $STATE <string>
## $ZIP <int64>
## $COUNTY <string>
## $PHONE <int64>
## $FAX_NUMBER <string>
## $WEBSITE <string>
## $LATITUDE <string>
## $LONGITUDE <string>
## $TOTAL_EMPLOYEES <int64>
## $EMPLOYEE_RANGE <string>
## $SALES_VOLUME <double>
## $SALES_VOLUME_RANGE <string>
## $CONTACT_FIRSTNAME <string>
## $CONTACT_LASTNAME <string>
## $CONTACT_FULLNAME <string>
## $CONTACT_GENDER <string>
## $CONTACT_TITLE <string>
## $CONTACT2_FIRSTNAME <string>
## $CONTACT2_LASTNAME <string>
## $CONTACT2_TITLE <string>
## $CONTACT2_GENDER <string>
## $NAICS_NUMBER <int64>
## $INDUSTRY <string>

toc()

## 39.862 sec elapsed

lobstr::obj_size(cleaned_arrow)

## 283.62 kB

Loading and Preparing the Bad Data to Stack

Next, we will attempt to also load the bad data into an arrow table, but will be unsuccessful (as shown), because there are still troublesome rows remaining, even after separating most of the good rows out, because some of the rows only have 19, while most have 22 columns.

bad_csv <- 
  try(arrow::read_csv_arrow(paste0(uscompany, "bad_scrub_data.csv")))

## Error in (function (file, delim = ",", quote = "\"", escape_double = TRUE,  : 
##   Invalid: CSV parse error: Expected 22 columns, got 19: "Ahlstrom, aaron",6282,FINANCIAL ADVISORY SERVICES,753 ameriprise financial ctr,Minneapolis,MN,5 ...

So, we go back to scrubscv and {data.table}, piping the command line function into fread(), and 1 million rows with the retained 22 columns takes about a second to load, throwing out 5,598 rows for having a non-standard number of columns. For this amount of data, it is hard to think of a better option!

cmd <- glue("scrubcsv { paste0(uscompany, 'bad_scrub_data.csv') }")
bad_data <- fread(cmd= cmd)

As mentioned previously, scrubcsv takes bad rows (ie: less than 22 columns in this case) and throws them out. It also doesn’t import column names with the discarded rows, so these have to be manually added after the fact, and since the table scan will be a significantly different amount of rows, it seems reasonable to expect some types may vary. We threw out the names of columns which were missing from bad_data, and then mapped to the appropriate column names in cleaned_arrow so that we could stack.

table_names <- names(cleaned_arrow)
names(bad_data) <-
  table_names[
    !table_names %in% c(
      "CONTACT2_FIRSTNAME",
      "CONTACT2_LASTNAME",
      "CONTACT2_TITLE",
      "CONTACT2_GENDER",
      "NAICS_NUMBER",
      "INDUSTRY"
    )]

For next part, we had quite a few false starts before we figured it out. {arrow} allows to bind rows, but column names have to match, and columns have the same data types. If we wanted to be picky, we might change some of the column types {arrow} chose for cleaned_arrow. For example, {arrow} chose Utf8 for the LON/LAT columns, which wasn’t what we expected, because they seemed to be numeric. For the definitive discussion of {arrow} data types though, please see Data types in Arrow and R. It is possible to convert data types in arrow tables, but it was not nearly as straightforward for us as doing it with {data.table} before converting to {arrow}.

# Convert integers, character and numeric to align with arrow types in cleaned_arrow
convert_ints <- 
  c("SIC_CODE", "ZIP", "TOTAL_EMPLOYEES")
bad_data[
  , (convert_ints) := lapply(.SD, bit64::as.integer64)
  , .SDcols = convert_ints]
convert_utf8 <-
  c("CONTACT_FULLNAME", "CONTACT_GENDER", "LONGITUDE", "LATITUDE")
bad_data[
  , (convert_utf8) := lapply(.SD, as.character)
  , .SDcols = convert_utf8]
bad_data[, SALES_VOLUME := as.numeric(SALES_VOLUME)]

After trial and error, we learned that by mapping R integer types to {arrow} int64, character to Utf8 and numeric to double, everything worked, and the conversion from {data.table} to {arrow} takes only an instant, so again timing is not shown.

# Convert to arrow table
bad_data_arrow <- 
  arrow::as_arrow_table(bad_data)
bad_data_arrow

## Table
## 1070764 rows x 22 columns
## $COMPANY_NAME <string>
## $SIC_CODE <int64>
## $SIC_DESCRIPTION <string>
## $ADDRESS <string>
## $CITY <string>
## $STATE <string>
## $ZIP <int64>
## $COUNTY <string>
## $PHONE <int64>
## $FAX_NUMBER <string>
## $WEBSITE <string>
## $LATITUDE <string>
## $LONGITUDE <string>
## $TOTAL_EMPLOYEES <int64>
## $EMPLOYEE_RANGE <string>
## $SALES_VOLUME <double>
## $SALES_VOLUME_RANGE <string>
## $CONTACT_FIRSTNAME <string>
## $CONTACT_LASTNAME <string>
## $CONTACT_FULLNAME <string>
## $CONTACT_GENDER <string>
## $CONTACT_TITLE <string>
## 
## See $metadata for additional Schema metadata

At first, we thought we might have to add the 6 missing columns and set the column order of bad_data_arrow to match those in cleaned_arrow, but it seems to work without adjusting column order or instructions to fill empty rows. Stacking the two data sets only takes an instant (so we are again not showing timing), and gives almost the full 31 million row data set we originally set out to load.

# Clean up and bind arrow tables
full_data <- 
  arrow::concat_tables(cleaned_arrow, bad_data_arrow)
full_data

## Table
## 31337290 rows x 28 columns
## $COMPANY_NAME <string>
## $SIC_CODE <int64>
## $SIC_DESCRIPTION <string>
## $ADDRESS <string>
## $CITY <string>
## $STATE <string>
## $ZIP <int64>
## $COUNTY <string>
## $PHONE <int64>
## $FAX_NUMBER <string>
## $WEBSITE <string>
## $LATITUDE <string>
## $LONGITUDE <string>
## $TOTAL_EMPLOYEES <int64>
## $EMPLOYEE_RANGE <string>
## $SALES_VOLUME <double>
## $SALES_VOLUME_RANGE <string>
## $CONTACT_FIRSTNAME <string>
## $CONTACT_LASTNAME <string>
## $CONTACT_FULLNAME <string>
## $CONTACT_GENDER <string>
## $CONTACT_TITLE <string>
## $CONTACT2_FIRSTNAME <string>
## $CONTACT2_LASTNAME <string>
## $CONTACT2_TITLE <string>
## $CONTACT2_GENDER <string>
## $NAICS_NUMBER <int64>
## $INDUSTRY <string>

Test Query and Benchmarking

It is amazing to get around all of the memory problems so easily, just by converting to an {arrow} table, but it doesn’t take long to then become greedy to for efficient data manipulation. To put it through the paces, we set up a test query to filter unique COMPANY_NAME on an aggregating variable (STATE or SIC_DESCRIPTION), count the number of occurrences, arrange in descending count order, filter the top 10 values (see hidden code below) and collect back into an R data.frame. STATE has only 51, but SIC_DESCRIPTION has 8,665 distinct values so should be a bigger lift to aggregate. We had heard by simply plugging in {duckdb} to_duckdb() into our {dplyr} chain, we might improve the performance of the query, so have included an option for that in our benchmark examples below. Below we show the query run once with just the {arrow} table (at around 10 seconds), which is substantially slower than the average of the same query, once we run it 100 times in our benchmarks.

See code

# Sample duckdb/arrow query function
test_agg <- function(data, agg_var, duck = FALSE) {
  
  if ( isTRUE(duck) ) {
    data <- data |> to_duckdb()
  }
  
  data |>
    select( {{agg_var}}, COMPANY_NAME) |>
    group_by({{agg_var}}) |>
    distinct(COMPANY_NAME) |>
    ungroup() %>%
    group_by({{agg_var}}) |>
    summarize(n = n()) |>
    ungroup() |>
    arrange(desc(n)) |>
    head(10) |>
    collect()
}

# Test run of query function
tic()
test_agg(full_data, agg_var=STATE)

## # A tibble: 10 × 2
##    STATE       n
##    <chr>   <int>
##  1 CA    2815269
##  2 FL    2022421
##  3 TX    1949658
##  4 NY    1498688
##  5 PA     917881
##  6 IL     914293
##  7 MI     755170
##  8 NC     732892
##  9 NJ     730344
## 10 VA     617868

toc()

## 10.231 sec elapsed

So here, we benchmark four queries, aggregating on STATE and SIC_DESCRIPTION with and without duckdb. The big surprise here was how big an impact {duckdb} had with so little effort, reducing query time by 55-60%. It also seemed to make the query run with much less variability, but will leave that to the experts to explain.

See code

# Microbenchmark on 100 iterations
mbm <- microbenchmark::microbenchmark(
  "state_arrow" = 
    test_agg(full_data, agg_var = STATE),
  "sic_arrow" = 
    test_agg(full_data, agg_var = SIC_DESCRIPTION),
  "state_duck" = 
    test_agg(full_data, agg_var = STATE, duck=TRUE),
  "sic_duck" = 
    test_agg(full_data, agg_var = SIC_DESCRIPTION, duck=TRUE)
)
mbm

## Unit: seconds
##         expr      min       lq     mean   median       uq       max neval  cld
##  state_arrow 6.039366 6.207929 6.411865 6.332908 6.527128  7.861368   100   c 
##    sic_arrow 8.025354 8.423692 8.827753 8.684798 9.171409 10.714919   100    d
##   state_duck 2.610817 2.703065 2.875179 2.800053 2.930623  3.848137   100 a   
##     sic_duck 3.275613 3.376994 3.812775 3.474520 3.824492  6.103654   100  b

## Coordinate system already present. Adding new coordinate system, which will
## replace the existing one.

When we were playing around, it seemed like the first time we ran a query was slower than after a few times. It seems like there might be a cost to moving over to duckdb, but we didn’t know how that would work. Looking at the time series of the queries, it looks like the first query was often slower, and then have big spikes in volatility after a while. Possibly not surprising, the larger group aggregation (SIC_DESCRIPTION) was more volatile, but it seems clear that {duckdb} makes query time more consistent.

See code

# Over time
mbm1 <- as.data.table(mbm)
mbm1[, trial := rowidv(mbm1, cols="expr")]
p <- ggplot2::ggplot(mbm1,
  aes(
    x = trial,
    y = time,
    group = factor(expr),
    color = factor(expr)
  )) +
  geom_line() +
  scale_y_continuous(
    labels = scales::label_number(scale = 1e-9)) +
  labs(x = "Trial",
       y = "Time [seconds]")

Conclusion

Based on this analysis, {arrow} offers a big jump in flexibility around the kind of analysis which can be conducted, seamlessly with the same work flow, from a small machine. The ease with which one line of code (without fiddling with any parameters), {duckdb} significantly reduced query time, justifying for us a lot of the raving on Twitter. Possibly more excitement about {arrow} and the way it dovetails with everything RStudio has already created may warranted. We haven’t shown here, but also saved the data as parquet, and ran queries against it in the {duckdb} CLI with read_parquet(), and got the sense that responses were even faster (despite whatever ingestion time was needed), but maybe that may be for a future post. We cannot express enough gratitude to RStudio, and to all the people who have developed {arrow} and {duckdb} at this breakneck speed

Loading a large, messy csv using data.table fread with cli tools

Thu, 21 Apr 2022 00:00:00 +0000

Setup

library(data.table)
library(here)

## here() starts at /Users/davidlucey/Desktop/David/Projects/redwall-analytics

library(glue)

## Warning: package 'glue' was built under R version 4.1.2

library(tictoc)
setDTthreads(percent = 90)
path_to_data <- "~/Desktop/David/Projects/uscompanies/data"
path_to_original <- here::here(path_to_data, "uscompanieslist.csv")

Introduction

On a recent side project, we encountered a large (7GB) csv of 30+ million US business names and addresses, which couldn’t be loaded into R, because of corrupted records. While not widely discussed, we have known for some time that it was possible to pipe command line instructions into {data.table}’s fread() by using its “cmd” parameter. However, there were only a few snippets available about how to do this, and most of these were constrained to limited strategies using awk. There were a few times in the past that we used awk, and we sometimes even got it to work, though we often didn’t understand why. awk seems like a great tool, but is like learning an entirely new language.

When we discovered Jeroen Janssens’ Data Science at the Command Line a few months ago, we realized there were a lot more possibilities for solving problems like this one. This book helped us to understand that the strategy of using fread()’s cmd capability might be expanded beyond awk. Unfortunately, the data set does not belong to us, so we cannot share it, but we will demonstrate the methods in case helpful for others.

Tools and Setup

The specific tools we been learning are {xsv}, {rg} (ripgrep), csvkit and scrubcsv. The first two were developed by BurntSushi using Rust, {csvkit} is a Python package, and {scrubcsv} is another Rust package inspired by the first two. We quickly learned that this tool set is a lot easier to install on Mac than Windows (using WSL), because most can be installed with Homebrew, the Mac package manager. We were not able to figure out how to install {xsv} and ripgrep on WSL, but “brew install xsv” and “brew install ripgrep” easily installed the libraries on our Mac.

Since we started our data journey about 5 years ago, managing Python installations has always been a challenge, and we will not discuss this here. Once Python is set up, the third is easy with “pip install csvkit”. Lastly, {scrubcsv} requires one step further, because there is no Homebrew formula, so first Rust and its package manager cargo had to be installed, which again can be accomplished with Homebrew following these instructions. Once installed, {scrubcsv} only requires “cargo install scrubcsv”.

Of the tools, {rg} is grep on steroids, while {xsv} and {csvkit} have many similar capabilities to slice and dice a csv. Though {xsv} is a significantly faster, {csvkit} has a built in cleancsv capability which can be used to solve our problem. {scrubcsv} does only one thing, it drops rows with the wrong number of columns, and it does this very fast. This seems like a more limited solution, but in our case it turns out to be just the ticket.

The Problem

As shown below, when we try to load the data set, we get “Error in fread(”/Users/davidlucey/Desktop/David/Projects/uscompanies/data/uscompanieslist.csv”, : R character strings are limited to 2^31-1 bytes”. We were not the only ones who have encountered this cryptic error, but it seemed the main way to solve it as outlined in this SO post https://stackoverflow.com/questions/68075990/loading-csv-with-fread-stops-because-of-to-large-string, is to ask the owner to reformat it, which wasn’t an option.

# Unsuccessful code
try(fread(path_to_original))

## Error in fread(path_to_original) : 
##   R character strings are limited to 2^31-1 bytes

As the SO poster was asking, it would be nice to be able to instruct fread() to try each and skip the bad rows, but this is not possible (at least from what we have figured out so far). We didn’t know which or how many rows specifically were causing the problem. Since the data set was so large, finding the problem, rows felt like a needle in a haystack, and the usual solution of loading it all into memory and looking around wasn’t possible.

Using `csvclean`

Like many who were previously scared by the CLI, the first step was to get over the fear of the help manual, cleancsv shown below.

{csvclean} Manual

csvclean -h

## usage: csvclean [-h] [-d DELIMITER] [-t] [-q QUOTECHAR] [-u {0,1,2,3}] [-b]
##                 [-p ESCAPECHAR] [-z FIELD_SIZE_LIMIT] [-e ENCODING] [-S] [-H]
##                 [-K SKIP_LINES] [-v] [-l] [--zero] [-V] [-n]
##                 [FILE]
## 
## Fix common errors in a CSV file.
## 
## positional arguments:
##   FILE                  The CSV file to operate on. If omitted, will accept
##                         input as piped data via STDIN.
## 
## optional arguments:
##   -h, --help            show this help message and exit
##   -d DELIMITER, --delimiter DELIMITER
##                         Delimiting character of the input CSV file.
##   -t, --tabs            Specify that the input CSV file is delimited with
##                         tabs. Overrides "-d".
##   -q QUOTECHAR, --quotechar QUOTECHAR
##                         Character used to quote strings in the input CSV file.
##   -u {0,1,2,3}, --quoting {0,1,2,3}
##                         Quoting style used in the input CSV file. 0 = Quote
##                         Minimal, 1 = Quote All, 2 = Quote Non-numeric, 3 =
##                         Quote None.
##   -b, --no-doublequote  Whether or not double quotes are doubled in the input
##                         CSV file.
##   -p ESCAPECHAR, --escapechar ESCAPECHAR
##                         Character used to escape the delimiter if --quoting 3
##                         ("Quote None") is specified and to escape the
##                         QUOTECHAR if --no-doublequote is specified.
##   -z FIELD_SIZE_LIMIT, --maxfieldsize FIELD_SIZE_LIMIT
##                         Maximum length of a single field in the input CSV
##                         file.
##   -e ENCODING, --encoding ENCODING
##                         Specify the encoding of the input CSV file.
##   -S, --skipinitialspace
##                         Ignore whitespace immediately following the delimiter.
##   -H, --no-header-row   Specify that the input CSV file has no header row.
##                         Will create default headers (a,b,c,...).
##   -K SKIP_LINES, --skip-lines SKIP_LINES
##                         Specify the number of initial lines to skip before the
##                         header row (e.g. comments, copyright notices, empty
##                         rows).
##   -v, --verbose         Print detailed tracebacks when errors occur.
##   -l, --linenumbers     Insert a column of line numbers at the front of the
##                         output. Useful when piping to grep or as a simple
##                         primary key.
##   --zero                When interpreting or displaying column numbers, use
##                         zero-based numbering instead of the default 1-based
##                         numbering.
##   -V, --version         Display version information and exit.
##   -n, --dry-run         Do not create output files. Information about what
##                         would have been done will be printed to STDERR.

As we discovered is often the case with UNIX tools, there were not as many walk-through detailed examples of {csvkit} as with many R packages. We found this one particularly cryptic as it seemed unclear about its output, but in hindsight, the -n command mentions “output files” which are created. We were concerned that it might alter our data, so created a backup and ran against that.

# Run previously to verify working, output on disc
system(command = glue::glue("csvclean {path_to_original}"))

After about an hour, the final output gives two new csv’s (“uscompanieslist_err.csv” and “uscompanieslist_out.csv”) by default, and leaves the original intact (uscompanieslist.csv). This is good, but means there is a need for a lot of disc space.

## [1] "uscompanieslist_err.csv" "uscompanieslist_out.csv"
## [3] "uscompanieslist.csv"

Bad Rows

In “uscompanieslist_err.csv”, csvclean adds two columns, one of which specifies the actual number of rows versus the number expected in each row. It also contains the line number of the original file where the problem was happening, which would have been nice to have earlier while we were hunting for bad rows. The cadence of our bad rows, which is every few thousand, can be seen and why our efforts at trying to load in chunks was problematic (chunks of a few thousand rows in 30 million).

Load uscompanieslist_err.csv Metadata

data <- 
  fread(here::here(path_to_data, "uscompanieslist_err.csv"), select = 1:2, nrows = 10)
data

##     line_number                                   msg
##  1:        5554 Expected 28 columns, found 22 columns
##  2:        5593 Expected 28 columns, found 22 columns
##  3:        5594 Expected 28 columns, found 22 columns
##  4:        8150 Expected 28 columns, found 22 columns
##  5:        8151 Expected 28 columns, found 22 columns
##  6:        8152 Expected 28 columns, found 22 columns
##  7:        8153 Expected 28 columns, found 22 columns
##  8:        8154 Expected 28 columns, found 22 columns
##  9:        8155 Expected 28 columns, found 22 columns
## 10:        8156 Expected 28 columns, found 22 columns

This file still contains rows with a differing number of columns, so still cannot be read by fread(). Here we use {rg} to filter out the remaining bad rows and {xsv} to drop the csvclean’s metadata columns, piped into fread(). In our case, most of the intact rows have 22 columns, instead of the expected 28, so we are guessing this data was somehow tacked on from another source. Although we use {rg} again here, we could have used grep and it probably wouldn’t have been much difference for 1 million rows, but it could also be done with any of the other tools or even with a traditional grep, also in about 10 seconds.

tic()
# Load bad_csvkit_data
bad_csvkit_data <- 
  fread(cmd = glue::glue(
    "rg '22 columns' { here::here(path_to_data, 'uscompanieslist_err.csv') } | xsv select 3-13,17,19,20-21,24"))
toc()

## 48.651 sec elapsed

As shown above there are 1070764 in the data set, and column names are lost and have to be manually re-inserted. At first, we were worries that the columns would be badly formatted, mistakenly merging columns, but looking at random samples of rows, this was not the case. A faster alternative with {scrubcsv}. There are also several columns which are missing all data or almost all blank cells. We can also add NULL columns for the ones which are missing.

Add Table Names

# Column names
data_names <- c(
    "COMPANY_NAME",
    "SIC_CODE",
    "SIC_DESCRIPTION",
    "ADDRESS",
    "CITY",
    "STATE",
    "ZIP",
    "COUNTY",
    "PHONE",
    "FAX_NUMBER",
    "WEBSITE",
    "EMPLOYEE_RANGE",
    "SALES_VOLUME_RANGE",
    "CONTACT_FIRSTNAME",
    "CONTACT_LASTNAME",
    "CONTACT_TITLE"
  )
names(bad_csvkit_data) <- data_names
sample <- bad_csvkit_data[sample(5)]

# Quick view of final data
sample

##              COMPANY_NAME SIC_CODE       SIC_DESCRIPTION              ADDRESS
## 1: Abdelbaki, zoheir a md     8011 PHYSICIANS & SURGEONS  770 w high st # 370
## 2:        Ackman, carmela     8111             Attorneys 110 e 42nd st # 1401
## 3:        Ackman, carmela     8111             Attorneys 110 e 42nd st # 1401
## 4:     Abel-hatzel, wendy     6411             Insurance          po box 1780
## 5:     Abel-hatzel, wendy     6411             Insurance          po box 1780
##        CITY STATE   ZIP   COUNTY      PHONE FAX_NUMBER        WEBSITE
## 1:     Lima    OH 45801    Allen 4192264310                          
## 2: NEW YORK    NY 10017 New York 2122531560            ackmanziff.com
## 3: NEW YORK    NY 10017 New York 2122531560            ackmanziff.com
## 4: COOS BAY    OR 97420     Coos 5412674124                          
## 5: COOS BAY    OR 97420     Coos 5412674124                          
##    EMPLOYEE_RANGE    SALES_VOLUME_RANGE CONTACT_FIRSTNAME CONTACT_LASTNAME
## 1:        1 to 10 $500,000 - $1,000,000           Shaheen            Abdel
## 2:        1 to 10   $100,000 - $500,000             Caryn           Effron
## 3:        1 to 10   $100,000 - $500,000              Alan          Goodkin
## 4:        1 to 10   $100,000 - $500,000           Harry D          Abel Jr
## 5:        1 to 10   $100,000 - $500,000        Wendy Abel           Hatzel
##        CONTACT_TITLE
## 1:           Manager
## 2:   Senior Director
## 3: Managing Director
## 4:   Insurance Agent
## 5:   Insurance Agent

As we mentioned above, csvclean took about an hour to run, but there are probably many other ways to accomplish our goal. Although we didn’t know the exact problem when we first tried csvclean, with hindsight, a better solution would have been {scrubcsv}, because it drops the rows with a differing number of columns, and it does so very quickly. One missing feature of {scrubcsv} is the lack of an output for the bad rows, so we had to capture these in the second line using the CLI comm command. In order not to fill up my disc further, these are not run here, but the total time to run both processes was just 5 minutes, and with a little cleaning, yields the same csv’s as {csvkit}, which took an hour.

CLI to Replicate {csvclean} with {scrubcsv}

Like the bad_csvkit_data, the output of bad_scrub_data still has a few rows with the wrong number of columns, but those are easily dropped with another run of csvscrub (shown in code chunk below) to remove all of the rows which do not have the predominant 22 columns, and using {xsv}, we also drop empty columns with {xsv} select.

Load bad_scrub_data

# Filter, select and load with fread
bad_scrub_data <- 
  fread(cmd = glue::glue("scrubcsv {path_to_data}/bad_scrub_data.csv | xsv select 1-11,15,17-19,22"))

# Use same names
names(bad_scrub_data) <- data_names

We can see that the output of the bad rows from the two methods are the same..

# Check if identical
identical(bad_csvkit_data, bad_scrub_data)

## [1] TRUE

Further Explorations

Here we show off a few tricks, with this trick scanning to locate Connecticut businesses from the 30 million rows in less than a minute. For example, we are able to stack the two data sets, filter the State of Connecticut and calculate the number of businesses by city. We would have liked to call the output from `fread()`, but in this case, the sub-processes from stacking the two tables seem to not be able to find the file paths from within R, so that is the first example of something which doesn’t work.

time xsv cat rows <(xsv select 1,5,6 ~/Desktop/David/Projects/uscompanies/data/scubbed_data.csv) <(xsv select 1,5,6 ~/Desktop/David/Projects/uscompanies/data/bad_scrub_data.csv) | xsv search -s STATE 'CT' | xsv frequency -s CITY

## CSV error: record 24 (line: 25, byte: 4189): found record with 19 fields, but the previous record has 22 fields
## field,value,count
## CITY,Stamford,22620
## CITY,Hartford,21278
## CITY,Norwalk,15085
## CITY,New Haven,14792
## CITY,Bridgeport,12111
## CITY,Danbury,10984
## CITY,Milford,10770
## CITY,Waterbury,9180
## CITY,Greenwich,8710
## CITY,Fairfield,8624
## 
## real 0m35.693s
## user 0m47.741s
## sys  0m6.376s

We can count the top 10 most states occurring in the data using xsv frequency and choosing the STATE column, which takes about a minute. The count seem roughly as expected, but a business in this data set can range from a sole proprietor to a multi-national. What we are really seeing is the number of locations which are a business.

tic()
data <- 
  fread(cmd = glue::glue('xsv select STATE {path_to_data}/scubbed_data.csv | xsv frequency'))
toc()

## 35.537 sec elapsed

data

##     field value   count
##  1: STATE    CA 3605240
##  2: STATE    TX 2584658
##  3: STATE    FL 2468792
##  4: STATE    NY 1972894
##  5: STATE    PA 1227555
##  6: STATE    IL 1221124
##  7: STATE    MI  967717
##  8: STATE    NC  945014
##  9: STATE    NJ  930482
## 10: STATE    VA  798290

For a grand finale, we thought it might be nice to find unique rows, but interestingly, we couldn’t find this as a built in capability in either {xsv} or {csvkit}, though both have requests to add it. The traditional sort | uniq would be pretty slow for such a large data set on our small computer, so we found another Rust library {huniq}. Now in the hang of it, there are so many resources available. It looks like if looked at by zip, it took about a minute to find out that there are 26 million unique businesses in the stacked data set, less than the full listed 31 million.

time xsv cat rows <(xsv select 1,7 ~/Desktop/David/Projects/uscompanies/data/scubbed_data.csv) <(xsv select 1,7 ~/Desktop/David/Projects/uscompanies/data/bad_scrub_data.csv) | huniq | xsv count

## CSV error: record 24 (line: 25, byte: 4189): found record with 19 fields, but the previous record has 22 fields
## 26431218
## 
## real 0m54.845s
## user 1m16.267s
## sys  0m55.631s

Conclusion

R is so often knocked for being slow, but views as wrapper of other tools like the Rust libraries, it might not be so true. {xsv}, {rg} and {huniq} were not as hard for us to understand as awk and surely perform a lot better. This exercise improved our confidence with the command line, and the tricks from Data Science at the Command Line. After a while referring to the man(ual) or help pages made, along with the usual Google search and Stack Overflow, we were able to figure out most challenges. Combined with fread(), it really starts to seem like a superpower at least with large, messy data sets. We are hoping that connecting the dots here will help others to solve similar problems.

Exploring Stock Market Listing Mortality since 1986

Sun, 29 Aug 2021 00:00:00 +0000

Click to see R set-up code

# Libraries
if(!require("pacman")) {
  install.packages("pacman")
}
pacman::p_load(
  data.table,
  re2,
  scales,
  ggplot2,
  plotly, 
  DT,
  patchwork,
  survival,
  ggfortify,
  scales)

# Set knitr params
knitr::opts_chunk$set(
  comment = NA,
  fig.width = 12,
  fig.height = 8,
  out.width = '100%'
)

NOTE: The read time for this post is overstated because of the formatting of the Plotly code. There are ~2,500 words, so read time should be ~10 minutes.

Click to see R code generating plot

# Load function to plot dual y-axis plot
source("train_sec.R")

# Get data series from FRED
symbols <- c("CP", "GDP", "WASCUR")
start_date <- '1947-01-01'
end_date <- '2021-07-30'
quantmod::getSymbols(
  Symbols = symbols,
  src = "FRED",
  start_date = start_date,
  end_date = end_date
)

[1] "CP"     "GDP"    "WASCUR"

# Merge series and convert to dt
d <- as.data.table(merge(WASCUR/GDP, CP/GDP, join = "inner"))

# Build superimposed dual y-axis line plot
sec <- with(d, train_sec(CP, WASCUR))
p <- 
  ggplot(d, aes(index)) +
    geom_line(aes(y = CP),
              colour = "blue", 
              size = 1) +
    geom_line(aes(y = sec$fwd(WASCUR)),
              colour = "red", 
              size = 1) +
    scale_y_continuous(
      "Corporate Profits to GDP",
      labels = scales::percent,
      sec.axis = sec_axis(
        ~ sec$rev(.),
        name = "Compensation of Employees to GDP",
        labels = scales::percent)
    ) +
    scale_x_date(date_breaks = "10 years",
                 date_labels = "%Y") + 
    labs(title = "Labor vs Capital",
         x = "Year",
         caption = "Source: Lots of places") +
    theme_bw(base_size = 22)

Introduction

The rise in monopoly power particularly in big technology, but possibly, more broadly across the economy and stock market has been a growing topic of conversation in recent years. America’s Monopoly Problem Goes Way Beyond the Tech Giants by David Doyen, and Monopolies Are Distorting the Stock Market, by Kai Wu of Sparkline Capital, one of our favorite data-driven bloggers, are good examples of this thinking. In addition to data on industry concentration, Wu draws the link to discussions over stagnating real wages, and sustained higher profit margins of companies since 1980 in the chart above, which we have seen referenced frequently in the last few years.

When we first heard these arguments, we were skeptical because it seemed like companies had been listing on and departing from exchanges at an accelerating pace, due to globalization and technological change. Industries like media and retailing have seen upheaval, and the departures of many formidable companies, and financial services seems like it could be next. If anything, our perception was that it was becoming more difficult for many incumbents constrained by legacy skills, processes and assets. Many of new listings, usually growing out of the cloud, brought new services we didn’t even know we needed, solved problems not previously addressed or disrupted dominant existing competitors. Spotify, Zoom, Docusign, Shopify, Roblox, Square and Snowflake come to mind as giant companies created in the last 10 years. Even the FAANG stocks of the previous generation are relatively young companies in the context of historical blue chips, and each of these brought us new innovations often at the expense of incumbents.

While still not sure if we are convinced of the monopoly threat, we ran into a possibly related issues in our work analyzing financial statement “red flags”. When we tried to match historical financial statements to subsequent return histories, a surprisingly large number of companies had been de-listed and fallen from data feeds. Once we had acquired the history for most, it was apparent that median, and by an even greater degree, the mean returns of companies trailed the Vanguard TMI (as shown in our Redwall Red Flag Explorer. We would have expected the average stock to do about as well as the collective members of the index, but that has not been the case, suggesting that success has been narrowing to a smaller group of larger enterprises.

How can the creation of promising young digital competitors and shortening lifespans of the average company be squared with the growing monopoly problem narrative? In this post, we will use Sharadar’s coverage to explore stock market listing births and deaths over the measured period. In light of the discourse, we hope to show in this brief post that the shortening of lifespans is undeniable.

Potential for Survivor Analysis with Sharadar Coverage

As discussed in When Yahoo Finance doesn’t have de-listed tickers needed, we chose to go with EOD Historical Data (Disclosure: we get a small credit for referrals), because our needs for now were only one-time, so the cost was substantially lower than the alternatives. However, the coverage spreadsheet of another provider, Sharadar, enabled us to tell exactly which tickers on our list were available in their database prior to investing time to figure out how to navigate the API. Coupled with data manipulation skills, the spreadsheet is a treasure trove of the history of 12,500 listed companies over 35 years, offering the option to explore when most new listings were created and often departed exchanges over several market cycles. Actuaries measure the life expectancy of people and data scientists of customers or website users, so now market watchers can measure the lifespan of an average investment.

An example of the data, we will use is shown in Figure @ref(fig:sharadar-ticker) for a single company. The firstpricedate will most often be when the company listed after going public or spinning off from another, The lastpricedate would generally be when it was purchased by another public or private company, investor or financial sponsor, or otherwise went out of business or stopped being the same legal entity, often though things like tax inversion. We are using the analogy of mortality, but it isn’t the business itself or its assets, but the stock market listing, which we are using as a representation.

Click to see R code loading data

# Load data for US and ADR common stocks only with data.table fread
path <- "~/Desktop/David/Projects/new_constructs_targets/"
categories <-
  c(
    "ADR Common Stock",
    "ADR Common Stock Primary Class",
    "Canadian Common Stock",
    "Domestic Common Stock",
    "Domestic Common Stock Primary Class"
  )
sharadar <-
  fread(paste0(path, "data/sharadar_tickers.csv"),
        select = c("permaticker", "category", "sector", "firstpricedate", "lastpricedate", "table", "sicindustry", "scalemarketcap"))
sharadar <- sharadar[
  table == "SEP" & category %chin% categories]

sharadar[permaticker == 122827]

   permaticker              category             sector firstpricedate
1:      122827 Domestic Common Stock Financial Services     1998-09-25
   lastpricedate table            sicindustry scalemarketcap
1:    2003-01-28   SEP State Commercial Banks      2 - Micro

Exploring Security Life and Death with Sharadar’s Coverage

We decided to divide listings into 5 cohorts by firstpricedate: initially listed prior to 1986, between 1986-2001, 2002-2008, 2009-2020, or still alive after that. As shown in Figure @ref(fig:birth-death-table), we noticed that Sharadar’s data shows no de-listings of any companies between 1986 and 1998. In the average year during the 1990s, hundreds tickers were de-listed each year. If that rate was typical, it implies we are probably missing data for thousands tickers which were de-listed between 1986-1998. Based on what we have learned about the difficulty of finding this information, we doubt that any of the providers would have complete information this far back.

Click to see R table code

# Create first, second and period variables
sharadar[
    , `:=`(
      first = fifelse(
        firstpricedate == "1986-01-01",
        1985,
        year(firstpricedate)), 
      last = fifelse(
        lastpricedate > "2021-06-01",
        2022,
        year(lastpricedate))
    )]

# Generate birth and death periods
sharadar[, `:=`(
  birth_period = fcase(
    first == 1985,
    "Pre-1986",
    first %in% c(1986:2000),
    "1986-2000",
    first %in% c(2001:2008),
    "2001-2008",
    first %in% c(2009:2021),
    "2009-2021",
    default = "missing"
  ),
  death_period = fcase(
    last %in% c(1986:2000),
    "1986-2000",
    last %in% c(2001:2008),
    "2001-2008",
    last %in% c(2009:2021),
    "2009-2021",
    last == 2022,
    "living"
  ))]

# Convert to birth and death tables by period and category (domestic or ADR)
births <-
  sharadar[, .N, 
    .(domestic = re2_detect(category, "Domestic"), 
      period = birth_period)][
        , dcast(
          .SD,
          period ~ domestic,
          fun.aggregate = identity,
          fill = 0)]
setnames(births, c("period", "ADR", "Domestic"))
deaths <-
  sharadar[, .N, 
    .(domestic = re2_detect(category, "Domestic"), 
      period = death_period)][
        , dcast(
          .SD,
          period ~ domestic,
          fun.aggregate = identity,
          fill = 0)]
setnames(deaths, c("period", "ADR", "Domestic"))

# Join together on period
table_list <- list(births, deaths)
table_list <- lapply(table_list, as.data.table)
table_list <- lapply(table_list, setkey, "period")
table <-
  Reduce(function(table1, table2){
    unique_keys <-
      unique(c(table1[, period], table2[, period]))
    table2 <- table2[unique_keys]
    table1[table2, on = "period"]}, table_list)

# Sort by time, rename and reorder
to_ord <- c(4, 1, 2, 3, 5)
table <-
  setorder(table[
    , .r := order(to_ord)], .r)[
      , .r := NULL]
setnames(table,
         c(
           "period",
           "adr_births",
           "dom_births",
           "adr_deaths",
           "dom_deaths"
         ))
setcolorder(table,
            c(
              "period",
              "dom_births",
              "dom_deaths",
              "adr_births",
              "adr_deaths"
            ))

# Make datatable
table <-
  DT::datatable(
    table,
    rownames = FALSE,
    colnames =
      c(
        "Period",
        "Domestic Births",
        "Domestic Deaths",
        "ADR Births",
        "ADR Deaths"
      ),
    options =
      list(pageLength = 5,
           dom = 't')
  ) %>%
  DT::formatRound(
    columns = c(2:5),
    mark = ",",
    digits = 0)

In Figure @ref(fig:history-plot) below, the left panel shows the births of new companies still alive today by sector over time. We can see the high rate that new companies were created during the 1990s (over 500 per year), but also the recent spike driven by the SPAC boom beginning in 2019. We understand that companies have been staying private longer, but the recent surge of “Blank Check” companies shown in blue in the left panel of Figure @ref(fig:history-plot) below, though worrisome, still looks relatively small compared to the 1990s. The fact that new entities have been listed more slowly until recently may not be all bad in light of what we learned of the poor quality listings during the boom.

In addition to fewer companies being listed until recently, our perception has long been that companies are dying at a faster rate because of globalization and technology. As Kai Wu hypothesized in his blog post, many may also have been acquired and/or perished due to stronger global competition and technological advantages. In the second panel, we can see the decline in average lifespan of company listings in all sectors over the period. After giving thought, we realized that younger companies in the data would naturally contribute shorter life spans closer to the end of the measurement period, so this graphic gives a misleading picture. It turns out that there are methods called “survivor analysis” which allows us to see an apples-to-apples picture of the rate of death of companies by cohort. R has the {survivor} package which is built for survival analysis, although we are probably taking liberties using it for non-overlapping periods.

Click to see R plot code

# Build ggplot2 object facets
p <-
    sharadar[
      birth_period != "alive"
      , .(.N, as.integer(round(mean(last - first), digits = 0))),
      .(sector, first)][
        sector != ""
        , melt(.SD, id.vars = c("first", "sector"))][
          , ggplot(.SD, aes(first, value, group = sector, color = sector)) +
            geom_line() +
            facet_wrap( ~ variable, scale = "free_y", 
                        labeller = labeller(variable = 
        c("N" = "Number of Issues",
          "V2" = "Mean Lifespan"))) +
            labs(
              x = "Year"
            ) + 
            theme_bw()]

# Add label title
p$labels$colour <- p$labels$fill <- "Sector"

# Transform to plotly
p <- ggplotly(p)

Coding up Survivor Analysis using the Magic of `{data.table}`

The data we have from Sharadar is a flat table with one row per company including the first and last listing date. In order to prepare to model with {survivor}, we had to make a relatively complex transformation, creating one row for every company during every year of life. In order to do this, we used a {data.table} (in the first section of the hidden code chunk below), which created a list for every year of life for every ticker, and then “exploded” that list by ticker back into a flat table. We have heard people excited about the Python {Pandas} explode() function, which is dedicated to this purpose. However, we prefer to solve most data wrangling problems using the same generalized toolkit of stable {data.table} functions.

We then had to add the “status” variable denoting if the company was alive with 0, or 1 for the year of death. Sharadar coded companies alive before the beginning of a single date of “1986-01-01”, but all of these companies were not born on that exact date, so it was a place holder for companies born in earlier years. We coded the time period for these years as “1985”, and companies “2022” for companies still alive after June 1, 2021. In order to create the “status” variable, we used the {data.table} elegant .SD (subset of data) filtering. This involved taking the first and last indexed date out of .SD (a kind of an fluid subset of our larger data.frame) by ticker, all using indices within the same code chain.

The last piece of {data.table} code to explain is the final segment, which creates four periods and then a list column of the other data for each of those period. It then merges those lists back into a single data.table including the relevant period. In the {Tidyverse}, this would involve loading {Tidyr} and using its special nest() and unnest() functions for list columns. Again, here we are using the same generalized {data.table} functions instead of a loading another package.

The prevailing school of thought is that the {Tidyverse} is easier to understand and appropriate for smaller data, and that {data.table} has a more complicated syntax, which should be reserved for bigger data. There is a learning curve, but after a couple of years, we feel excitement when approaching this kind of complicated transformation to see the magic of {data.table} solve the problem. In addition to being fastest, {data.table} is also stable without frequent new, specialized functions and changing parameters. Trying to use it only on larger data would probably mean we couldn’t use it to its fullest extent. Finally, we would argue for its elegance over the {Tidyverse}, regardless of data size. After wrangling, the data shown below is for the same ticker we showed above. This company was born in the pre-1986 period and was de-listed within 5 years.

Click to see R code

# Create list of data.tables appropriate for survivor modelling
mortality <-
  sharadar[
      sicindustry != "Blank Checks",
    .(permaticker, alive = list(list(first:last))),
    .(first = 
        fifelse(
          firstpricedate != "1986-01-01", 
          year(firstpricedate), 
          1985),
      last = year(lastpricedate))][
        , list(year = unlist(alive)),
        by = permaticker][
          , {
            first = .SD[1]$year
            last = .SD[.N]$year
            time = year - first
            status = fcase(
              year == 2020 & last == 2020,
              1,
              year == 2021 & last == 2021,
              0,
              year < 2020 & year == last,
              1,
              default = 0
            )
            period = fcase(
              first == 1985,
              "Pre-86",
              first %in% c(1986:2000),
              "1986-2000",
              first %in% c(2001:2008),
              "2001-08",
              first %in% c(2009:2020),
              "2009-20",
              default = "missing"
            )
            list(time, status, period)
          }, permaticker][
            period != "missing",
            list(list(.SD)), period]

# Transform lists into a single data.table by period
mortality <- 
  rbindlist(mortality$V1,
            use.names = TRUE,
            fill = TRUE,
            idcol = "period")

# Display transformed mortality data for one company
mortality[permaticker == 122827]

   period permaticker time status
1:      1      122827    0      0
2:      1      122827    1      0
3:      1      122827    2      0
4:      1      122827    3      0
5:      1      122827    4      0
6:      1      122827    5      1

Building the Model and Plotly Object

In this section, we would also like to discuss some coding tricks we have only recently learned about {ggplot2} (which are also effective with {plotly} objects). Given that we have been using these packages so frequently for over four years, it is surprising that it took so long to learn this trick, so it might help others to demonstrate. The {survivor} package has a pre-built function in {ggfortify} to plot Kaplan-Meier (km) objects. The Kaplan-Meier estimate of the survival probability is the product of the conditional probabilities until the event (in this case de-listing). Ordinarily there are treatment groups in a study (ie: treatment and control), but we are pretending that each of the four time periods are separate cohorts in a study.

When we went to view the plot, we found that it was more complicated than we expected to change the legend labels inside the {ggplot2} chain. Thankfully, we can go inside these objects and manually make adjustments after they have been assigned. In the code block, we change the “strata” names to our “period” groups, and now when we run autoplot() on the km object, the labels are fixed. In a similar manner, we manually change the legend title for the color and fill attributes of the list object after we build the autoplot(). Then, the graph shows exactly as we intended.

Click to see R plot code

# Build km object
km <- 
  survfit(
    Surv(time = time, event = status) ~ period, 
    data = mortality)

# Change legend labels
names(km[["strata"]]) <-
  c("Pre-1987", "2009-2021", "2002-2008", "1987-2001")

# Build autoplot
p <- 
  autoplot(km) + 
  theme_bw() + 
  labs(
    title = "Annual Listing Survival Rates by Cohort",
    caption = "Source: Sharadar",
    x = "Age",
    y = "Survival Rate"
  )

# Modify legend title
p$labels$colour <- p$labels$fill <- "Period"

Thoughts on Survival Rates by Period

The top purple estimate (1987-2001) does not actually represent the earliest cohort (pre-1986), but the second in which companies started to be born. The 100% survival rate in the first 12 years of this cohort suggests that the data for de-listings may be missing (and hence also for initial listings). At the same time, we can also see with the red chart (pre-1986), that the rate survival during two early cohorts was surely higher than in the later two periods. The change in the survival rate by year visible in the size of the estimated vertical steps along the y-axis. Once companies start to show de-listings around 1997 (12 years into lives of the companies in that cohort), we can see that the survival rate declines by about 1-2% per year.

The different story starts with companies born in the 2002-2008 cohort, when the survival rate decline is much more rapid. We can see that the years of age are shorter, because every company in the period is measured from its first trading year, and confidence bands are wider, because there are fewer companies included in those cohorts. When the more recent bull market began in the 2009 cohort (green), the survival rate is even lower (even though we have actually removed the most recent surge in SPAC issuance). It also seems that the rate of de-listing for the earlier groups also accelerates around the same time as the later cohorts start. Looking at the x-axis from right to left allows to see comparable periods, and interestingly, the most recent cohort appears like it will soon catch up to the others, even though it started 12 years after then next latest. All in all, about 20% of companies for all of the cohorts have been de-listed.

Conclusion

We may have taken liberties using different time cohorts as treatment, but seemed like a better than the alternative of including them all in one group. Our analysis doesn’t help at all to understand the causes of or possible remedies for the shortening life of listings, or if it is even good or bad. Shortening lifespans of companies may be caused by the predatory behavior of monopolies, accelerating obsolescence of enterprises, Dodd-Frank raising the cost of listing, tax inversions, or some other combination of factors. Although as guilty as any of throwing up charts on this blog and speculating over what they might signify, we will not attempt to do so here. Suffice it to say that it listing lives have shortened, and we will leave it for others to explain the true causes.

When Yahoo Finance doesn't have de-listed tickers needed

Thu, 19 Aug 2021 00:00:00 +0000

Click to see R set-up code

# Libraries
if(!require("pacman")) {
  install.packages("pacman")
}
pacman::p_load(
  data.table
  )

# Set knitr params
knitr::opts_chunk$set(
  comment = NA,
  fig.width = 12,
  fig.height = 8,
  out.width = '100%'
)

Introduction

As we discussed in our last post Introducing the Redwall ‘Red Flag’ Explorer with New Constructs Data, we were able to test the response of 125,000 quarterly and annual financial statements to incidence of “red flag” ratios, but some of the most interesting results may have been hidden in de-listed tickers, often not available to the default R {quantmod} pricing sources (Yahoo Finance and Alpha Vantage). As a result, 15-25% of financial statements in New Constructs the first five years of data, could not be matched to return history, exposing our analysis to “survivor bias”. Redwall’s mission is multifaceted: to conduct research on topic of interest in available financial and government data, but also to explore and share how to best solve problems encountered while using R.

We would love to see market sponsors make weekly or monthly prices available to the open source community, as has become almost expected in so many other areas where open data is used. Even though every closing price was once publicly disclosed in the newspaper, that doesn’t mean that those prices are available for analysis, and it takes a lot of work to collect them. One of the only options suggested online, was to go to the local library and look through the stacks, but this was not feasible for a personal project involving over 900 securities. It turns out that there are several providers of financial statement data for a price. Since many other R users will likely face this same problem, one objective of this post will be to lay out the options for others looking for older and de-listed securities.

Financial Data API’s for De-Listed Securities

Just to make clear, this is not the first article about what to do when Yahoo Finance fails. In our case, we looked at four providers, the first being Sharadar, as above mentioned. Sharadar had a substantial listing of 16,512 tickers, going back to 1986 and including ADRs, Canadian and US companies. We discovered that we could match the large majority of our still-missing tickers, but the cost for the level of access allowing the full history needed would be a full year’s access at their recently reduced price of $299 (through the Quandl API). We also looked at Algoseek, but their data only went back to January 2007, so that wouldn’t solve the problem. Norgate offered access to de-listed securities back to 1990 for $346 under their 6-month “Platinum” plan option. Norgate also offered a free 3-week trial, but that only included 2 years of data, which would involve time to figure out the API and wouldn’t solve the problem, unless we went for the subscription. Lastly, there was EOD Historical Data (Disclosure: after we wrote this post, we were offered a small credit for referrals to EOD via this link), which offered $20 per month for world prices going back 30+ years.

Finding Matching Prices

Sharadar’s comprehensive spreadsheet, offering certainty that needed prices would actually be available without having to download through the API (a helpful feature while deciding how to proceed). Given that this is only a personal project, we thought we would try EOD first to see if we could access the data we needed for only $20, though this forced us to invest time to get the API working from R without knowing if what we were looking for would even be there.

EOD gave a code example for accessing the API, but we struggled at first to use it with the suggested default .csv format. The error messages were confusing, and sometimes unclear if the data we needed was not available (404) or if there was a problem with the API preventing the download. Customer support was very responsive, considering they are not charging a lot for it, and after some wheel spinning, helped us to understand that using JSON would work better. By adding “&fmt=json” to the set-up string and parsing with {jsonlite} as in the code example below worked perfectly, and we were able to collect over 800 of our 930 missing tickers.

As we were doing this, we also discovered other problems in our Yahoo Finance prices, so decided to collect the tickers we previously thought we would use from Yahoo. The All-World package allows up to 100,000 calls per day, which was more than enough, and we would be able to expand beyond the US to 60 overseas exchanges. EOD was felt quite a bit faster than the traditional {quantmod}, even when we used {BatchGetSymbols} for collecting a lot of tickers. Unlike with Alpha Vantage, these were adjusted prices. All in all, EOD seemed to be perfect for our use-case.

Click to see R code sample API call

# Tickers and API token
api.token <- "OeAFFmMliFG5orCUuwAKQ8l4WWFQ67YX"
tickers <- c("AAPL.US")
ticker_list <- list()

# Loop to collect prices
for ( ticker in tickers ) {
  
  # Set up API string
  ticker.link <-
    paste(
      "http://nonsecure.eodhistoricaldata.com/api/eod/",
      ticker,
      "?api_token=",
      api.token,
      "&period=w&order=d&fmt=json&from=1997-01-01",
      sep = ""
    )
  
  # Call to API
  ticker_prices <- try(jsonlite::fromJSON(ticker.link))
  
  # Wait to avoid overloading API
  Sys.sleep(2)
  
  # Append new ticker data to list
  ticker_list <-
    append(ticker_list, list(ticker_prices))
  
}

# Name list by ticker
names(ticker_list) <- tickers

ticker_list$AAPL.US[1:10,]

         date    open     high    low  close adjusted_close    volume
1  2021-08-23 148.310 150.8600 147.80 148.36       148.3600 166528933
2  2021-08-16 148.535 151.6800 144.50 148.19       148.1900 429022231
3  2021-08-09 146.200 149.4444 145.30 149.10       149.1000 299579344
4  2021-08-02 146.360 148.0450 145.18 146.14       146.1400 284559336
5  2021-07-26 148.270 149.8300 142.54 145.86       145.6418 423324004
6  2021-07-19 143.750 148.7177 141.67 148.56       148.3378 441563672
7  2021-07-12 146.210 150.0000 143.63 146.39       146.1710 504249353
8  2021-07-06 140.070 145.6500 140.07 145.11       144.8929 418559704
9  2021-06-28 133.410 140.0000 133.35 139.96       139.7506 321360121
10 2021-06-21 130.300 134.6400 129.21 133.11       132.9109 354155886

Some thoughts on the collected data

We collected pricing data for an additional 400 tickers than with our original sources, so we are still missing price histories for about 500. Of these, only about 200 had 10 or more filings, so many of these were short-lived listings, and possibly not as relevant for our analysis. We also learned that in cases where the ticker had been used more than once, EOD price history would generally have the most recent, but not for the previous entity. For example, “DELL” went private in 2014 and was re-listed in 2016, so only data for the latter entity was available with EOD. In these cases, we were often able to use the Yahoo data.

In addition, we learned that Yahoo sometimes gave surprising results. In the case if ticker “EDO” below (not to be mistaken with EOD who have provided the data), the reliability of the Yahoo data on the same dates during the early period, is shown to be very bad. Here we are showing for prices when there should be none, very small and unchanging prices in some periods, and then jumping all over the place when the history should have ended.

Click to see R function used to generate output

get_ticker <- function(ticker) {
  # Load prices for ticker from EOD and Yahoo
  path <- "~/Desktop/David/Projects/new_constructs_targets/data/"
  eod <-
    fread(cmd = paste0("grep ", ticker, " ", path, "eod_weekly_prices.csv"))
  setnames(eod, c("ticker", "date", "adjusted.close"))
  yahoo <-
    fread(cmd = paste0("grep ", ticker, " ", path, "historical_prices/nc_complete_prices2.csv"))
  yahoo <- yahoo[, c(1:2, 8)]
  setnames(yahoo, c("ticker", "date", "adjusted.close"))
  
  # Rbind, order by date and dcast for comparison
  prices <- list(eod, yahoo)
  names(prices) <- c("eod", "yahoo")
  prices <-
    rbindlist(prices, idcol = "source")[order(date)]
  
  # Return
  prices[
    , dcast(
      .SD, 
      date ~ source,
      fun = identity,
      fill = NA_real_)]
}

get_ticker("EDO")[c(1:10)]

          date   eod yahoo
 1: 1997-12-31 8.688    NA
 2: 1998-01-02    NA  0.51
 3: 1998-01-05 8.438  0.51
 4: 1998-01-12 8.625  0.51
 5: 1998-01-20 8.500  0.51
 6: 1998-01-26 9.000  0.51
 7: 1998-02-02 9.500  0.51
 8: 1998-02-09 8.938  0.51
 9: 1998-02-17 9.000  0.51
10: 1998-02-23 9.125  0.51

Here is another example of SKP illustrating the potential problems, considering our desire to find accurate return data after a given date. These price sequences would give a lot of cases of zero returns, and others with very positive or negative returns.

get_ticker("SKP")[c(51:60)]

          date    eod yahoo
 1: 1998-12-14     NA 20018
 2: 1998-12-21     NA 20018
 3: 1998-12-28     NA 20018
 4: 1999-01-04 28.875 20018
 5: 1999-01-11 28.500 20018
 6: 1999-01-19 28.625 20018
 7: 1999-01-25 29.688 20018
 8: 1999-02-01 29.063 20018
 9: 1999-02-08 29.125 20018
10: 1999-02-16 29.625 20018

For this reason, we favored EOD prices, followed by Alpha Vantage when not available, and finally Yahoo Finance as the last resort, which meant the large majority were coming from EOD as shown in the table below.

path <- "~/Desktop/David/Projects/redwall-analytics/content/post/2021-08-19-when-yahoo-finance-doesn-t-have-de-listed-tickers-needed/"
source(paste0(path, "prices_source_table.R"))
prices_source_table()

   source       N
1:    eod 3680589
2:  yahoo  267814
3:     av  406068

Now, we find that we are missing about 200 tickers out of the 5,500 we set out to match. Many of these were ending in “Q” (so were already on the pink sheets), are foreign or may have bad tickers. In any case, we have probably covered the bulk of the meaningful companies with our analysis.

  [1] "AABA"       "AAMRQ"      "AANI"       "ABCWQ"      "ABII"      
  [6] "ABKFQ"      "ABLSQ"      "ACHI"       "ACME_33987" "ACPIQ"     
 [11] "ACTT"       "AEMI"       "AFR"        "AGCCQ"      "AHMMQ"     
 [16] "AKRXQ"      "ALTV"       "AMOA"       "ANCCQ"      "ANLD"      
 [21] "ANRZQ"      "ANSVQ"      "ANVGQ"      "APAC"       "APXSQ"     
 [26] "ARDI"       "ARMP_1725"  "ASCL"       "ASYTQ"      "ATNY"      
 [31] "ATPAQ"      "AWA"        "BBAO"       "BCKDY"      "BFCF"      
 [36] "BHEL.BO"    "BKUNQ"      "BLGM"       "BOW"        "BRLC"      
 [41] "BSBN"       "CAMD"       "CBCG"       "CBSS"       "CCOWQ"     
 [46] "CDWC"       "CEMJQ"      "CHMT"       "CHZS_8434"  "CLK"       
 [51] "CNVX"       "COHM"       "CPGVY"      "CRGIY"      "CRNM"      
 [56] "CSKEQ"      "CTCI"       "CTGI"       "CTRA"       "CUNO"      
 [61] "CZZ"        "DCGNQ"      "DDIC"       "DEU.F"      "DHI.KS"    
 [66] "DIVX"       "DOLE"       "DRTE"       "EBHIQ"      "EGLSQ"     
 [71] "EKDKQ"      "EMRG"       "ENMC"       "EPEXQ"      "ERPLQ"     
 [76] "EZEM"       "FBNIQ"      "FCE.A"      "FCHDQ"      "FCSE"      
 [81] "FFKY"       "FILE"       "FNBP"       "FSNM"       "FWMHQ"     
 [86] "GCORE"      "GDPMQ"      "GDYS"       "GISX"       "GNKOQ"     
 [91] "GVHR"       "HECL.F"     "HEVV"       "HOFF"       "HRVE"      
 [96] "HSTN"       "HYDP"       "IBCPD"      "IION"       "IMNR"      
[101] "INFS"       "ISOOE"      "ITWO"       "JHTXQ"      "JRCO"      
[106] "JWLR"       "KNGGY"      "KVPHQ"      "LAWE"       "LCAV"      
[111] "LEHMQ"      "LFGRQ"      "LGFTY"      "LINEQ"      "LNX"       
[116] "LPHIQ"      "LPS"        "LQI"        "MCEL"       "MDRIQ_5781"
[121] "MECAQ"      "MHRCQ"      "MILLQ"      "MKTS"       "MODT"      
[126] "MSNW"       "MSSN"       "MTLQQ"      "MUSE"       "MYG"       
[131] "NCOC"       "NHR"        "NMGA"       "NNDS"       "NRVHQ"     
[136] "NUI"        "NVTP"       "NWACQ"      "NWEC"       "OLAB"      
[141] "OO"         "OSCIQ"      "PENX"       "PILLQ"      "PMRY"      
[146] "PRFS"       "PRLI"       "PRTLQ"      "PRXZ"       "QDHC"      
[151] "QRCP"       "RCOCQ"      "RCRC"       "REVUQ"      "RGAA"      
[156] "RGFC"       "RITA"       "RSTO"       "SBIT"       "SBLKE"     
[161] "SEN.ETR"    "SEPR"       "SGGHU"      "SHRPQ"      "SHZ"       
[166] "SKRRF"      "SMF"        "SPCBQ"      "SPNVD"      "SQAA"      
[171] "SRNAE"      "SUZ.DEU"    "SVNT"       "SZ"         "TARRQ"     
[176] "TCPTF"      "THQIQ"      "TLCR"       "TMG"        "TNSIQ"     
[181] "TOBC"       "TORCQ"      "TRMS"       "TWLL"       "TWTRQ"     
[186] "UNISQ"      "UPFC"       "USFC"       "USPI"       "UTA"       
[191] "UVSLQ"      "VARI"       "VIONQ"      "VLCY"       "VRAIE"     
[196] "VRLKQ"      "VRST"       "VSE"        "VTAI"       "VTRO"      
[201] "WAVXQ"      "WC"         "WFTIQ_8616" "WGATQ"      "WG.L"      
[206] "WGRD"       "WLS"        "WMANQ"      "WNNB"       "WPSL"      
[211] "WWCA"       "XYBRQ"      "YAKC"

Conclusion

In the end, our return data is still likely not perfect, but is pretty comprehensive, and certainly the data visualization reflected in our Red Flag App should be close to an unbiased reflection. Most of the four million stock returns we collected did not cost an unreasonable amount (as long as not used for commercial purposes). In our next post, we will explore the life and death of companies since 1986 using some of the data we collected in this project.

Introducing the Redwall 'Red Flag' Explorer with New Constructs Data

Mon, 09 Aug 2021 00:00:00 +0000

NOTE: The read time for this post is overstated because of the formatting of the code. There are about 4,400 words, so read time should be closer to 15 minutes.

Click to see R set-up code

# Libraries
if(!require("pacman")) {
  install.packages("pacman")
}
pacman::p_load(
  data.table,
  scales,
  ggplot2,
  plotly, 
  DT)

# Set knitr params
knitr::opts_chunk$set(
  comment = NA,
  fig.width = 12,
  fig.height = 8,
  out.width = '100%'
)

# Load annual data only
path <- 
  "~/Desktop/David/Projects/new_constructs_targets/_targets/objects/"
red_flags <- 
  readRDS(paste0(path, "nc_annual_red_flags"))
annual_data <- 
  readRDS(paste0(path, "nc_annual_final"))

Key Findings

1999-2000 was an exceptional period for both “Red Flag” prevalence and return differentiation, though apparent benefits of the strategy appear in most periods.
Approximately 2.5% of filings we checked had 5 or more “Red Flags” among annual and quarterly filings, so sparsity is challenge in estimating true relative returns.
Finding price histories for de-listed companies in open source channels is an challenge, which likely creates “survivor bias” in relative return estimates.
Prevalence of accounting distortion declined considerably since 1999-2000.

Introduction

A few months ago in A Blueprint of “Red Flag” alerts Using Adjusted Earnings Data, we mused about using New Constructs historical data to back-test ideas from a 2003 CFA Conference Proceedings article entitled Revelations from Financial Reporting by Bruce Gulliver. This article succinctly offered the theory that, in the aftermath of the collapse of “Dot-com Bubble”, impending losses for many stocks might have been avoided by using a simple set of financial statement ratios, collectively as “red flags”. Mr. Gulliver’s work always stuck with us, and especially now that R can easily be used to test the hypothesis and to take the analysis to a different scale and interactivity. The only other missing piece would be thousands of companies with consistently and meticulously adjusted financial statement data over a very long period, which as far as we know, only resides at New Constructs.

After our blog post, New Constructs kindly offered to let us use their data to test the theory on a large sample of companies and over the last twenty years, which includes two of the great stock market busts and the recent Covid-19 related volatility. In this post, we will describe our analysis, summarize the benefits of the unparalleled New Constructs data and the interactive Red Flag Explorer Shiny app we have built for anyone who would like to interact with our derived “red flags” and measure their performance. Other than being the beneficiary of their generous support for our project, we do not have any business relationship with New Constructs.

It is striking that analysis on this scale (ie: analyzing ~125,000 unique financial statements), impossible for a regular person with a computer to produce (much less share with others) for most of our years following markets, can now be conducted in a few weeks of coding. We think our “red flags” built on top of New Construct’s data represent a unique historical fingerprint of the market’s reaction to accounting disclosures in the controversial post-2000 period.

Red Flag Calculations

The methodology for generating “red flags” was close to what we laid out in A Blueprint of “Red Flag” alerts Using Adjusted Earnings Data, but there were several differences.

We didn’t fully understand the Revelations cash flow adjustments, which primarily had to do with timing differences of expensing of employee share options. According th Mr. Gulliver’s article, this became very significant during the Dot-com Bubble in many companies, but we are not sure if it is still so, and didn’t have comparable data from New Constructs to calculate it. If we were to do it again, we would have requested the data to calculate the difference between New Constructs “True Free Cash Flow” and “Traditional Free Cash Flow”, as laid out in The Most Overstated And Understated FCF In The S&P 500 Post 1Q21 Earnings by David Trainer (CEO of New Constructs) in the June 14th Forbes issue. For now, our app won’t have a “red flag” pertaining to cash flow.
Asset turnover was calibrated relative to other companies in the same sector, but considering the full 20+ year period when determining unfavorable ratios.
Trend-related variables (ie: increasing days of inventory/receivables, declining margins and ROIC and declining reserves/sales ratios) were calculated by taking more than one year of change into account to give an added penalty when the negative trend was persistent. This had a cost of losing the two periods at the very beginning of the series (ie: 1997-1998), because those were needed for the look-back.
The High Valuation “red flag” was calculated using New Construct’s three valuation-based Stock ratings (FCF Yield, Price-to-Economic Book Value and Market Implied Growth Appreciation Period), themselves derived variables, rather than raw valuation metrics. New Constructs does not use the traditional GAAP earnings and book value ratios pointed to in Revelations to determine valuation, for reasons very well discussed in the Basic Metrics section on New Construct’s website.
We don’t know what Mr. Gulliver would have used, but we defined “high” Earnings Distortion as an equal aggregation of the highest divergence of NC adjusted from reported net earnings divided by market capitalization and the absolute value of reported net earnings.
For liquidity, we first screened that a company did not have “excess cash”, a New Constructs derived variable measuring the amount of cash over and above what was needed to conduct operations. If the company did not have “excess cash”, we then used several credit metrics similar to those discussed in Revelations, but using the comparative New Construct’s derived financial statement items.
We also added two additional “red flags” of our own, for companies having amended filings (by far the least common “red flag”) and with more than two flags previously still showing filing-on-filing increases in total flags.

Thoughts on Red Flags

We don’t know what thresholds Mr. Gulliver would have used in his calculations, but where we had discretion, our “red flags” were calibrated to occur in about 20% of filings over the “whole period” (as shown in Figure 1 below). Because some flags were less frequent, the average rate of occurrence over all 10 flags was 13.8%. If the probability of raising flags was independent, this would translate into a 0.6% probability of having 5 or more flags (based on the binomial), but almost 2.5% of filings analyzed had that many flags, so this may be a sign that some flags contribute to the likelihood of others and may not be independent.

When we say “over the whole period” above, this is significant because it means that the cut-offs for a “red flag” is the same regardless of the reporting period. Another option would have been to calculate by period (ie: attributing a similar number of each red flag in each period), but that would have taken away the ability to compare behavior over time. There was no special knowledge consideration given to “informed” threshold levels, where evidence supported likely problems, just that the selected metric was deviating negatively relative to the large majority of filings during the 20-year period. Further work in this regard might even improve the quality of the signalling.

Click to see R plot code

# Melt on logical cols as measure
cols <- names(red_flags)[sapply(red_flags, is.logical)]

# Melt data to long on fiscal_year and total_flags
red_flags_long <-
  red_flags[
    data.table::between(fiscal_year, 1999, 2020),
    lapply(.SD, mean),
    .SDcols = cols,
    fiscal_year][
    ][order(fiscal_year)][
    ][, data.table::melt(.SD, measure.vars = cols)]

# Make basic ggplot on x = fiscal_year and y = total_flags.
p <- 
  red_flags_long[, 
    ggplot2::ggplot(
      .SD,
      ggplot2::aes(x = fiscal_year,
                   y = value,
                   color = variable,
                   fill = variable)) +
      ggplot2::geom_line() +
      ggplot2::geom_point(size = 1) +
      ggplot2::labs(
        x = "Fiscal Year",
        y = "Percentage of Filings") +
      ggplot2::scale_y_continuous(
        labels = scales::percent) +
      ggplot2::theme_bw()]

# Render as plotly and add customized flag labels to plotly object
p <- plotly::plotly_build(p)

names <- c(
  "Amend.",
  "Low Return",
  "Earns. Distort.",
  "Reserve Decline",
  "Days Inv or A/R ",
  "Mgn & ROIC decline",
  "Asset Turns",
  "High Val'n",
  "Poor Liquid.",
  "Neg. Trend")

vars <-
  c(
    "amended",
    "agg_returns",
    "aggregate_distortion",
    "reserves_indicator",
    "bs_indicator",
    "margins",
    "turnover_flag",
    "agg_rating",
    "liquidity",
    "trend"
  )

# Add red flag labels and tooltip to Plotly object
for (i in 1:10) {
  p$x$data[[i]]$name <- names[i]
  
  d <- 
    red_flags_long[variable == vars[i]]
  
  p$x$data[[i]]$text <- paste(
    "Period: ",
    d$fiscal_year,
    "<br>",
    "Red Flag Indicator: ",
    names[i],
    "<br>",
    "Percent of Occurrences: ",
    paste0(round(d$value * 100, 0), "%"),
    "<br>"
  )
}
p[["x"]][["layout"]][["annotations"]][[1]][["text"]] <- "Red Flag"

# Add "Source: New Constructs" to bottom right
p <- 
  p %>% plotly::layout(
    hoverlabel = list(align = "left"),
    annotations =
      list(
        x = 1.05,
        y = -0.10,
        text = "Source: New Constructs",
        showarrow = F,
        xref = 'paper',
        yref = 'paper',
        xanchor = 'right',
        yanchor = 'auto',
        xshift = 0,
        yshift = 0,
        font = list(
          size = 12,
          color = "darkgray")
      )
)

Figure 1: Prevalence of Most Red Flag Declined Since Early 2000s

Ex-Post Return Data

In order to calculate ex post quarterly returns, we tried to find matching weekly prices for every company in the New Constructs database using the R {BatchGetSymbols} package, a wrapper for {quantmod}, when more than a few hundred tickers are needed. Both packages source prices from Yahoo Finance by default, and provided price histories for almost 4,000 requested companies. About 1,700 tickers, generally defunct since the earlier periods of the series, were not available in Yahoo Finance, but we were able to recover an additional ~800 of the missing price histories using Alpha Vantage (the main pricing alternative to Yahoo Finance offered by {quantmod}), leaving ~900 companies unmatched.

While Yahoo maintains, and we used the “adjusted prices” (ie: for splits, dividends and other corporate actions) when available, Alpha Vantage only offered closing prices (un-adjusted). Although we assume that many of these must have gotten into difficulty and otherwise been de-listed, some might have been subsumed into other companies, possibly at a premium. If the stock price went to zero or was otherwise de-listed from trading, we think the fact that the price data was not adjusted might be less relevant, because it probably wasn’t paying dividends, spinning off subsidiaries or successfully completing a rights offering. Still, the 800 companies currently using closing prices from Alpha Vantage may cause some inaccuracy in our return estimates.

When we matched companies with returns, we used a “rolling join” on ticker and date, taking the last weekly price after to the filing date to simulate purchasing after the number of “red flags” was known. For this reason, the return calculated might start at the end of the same day as the report, or 1-4 days after the filing date. As a result, our relative return estimate should tend to be conservative approximation of the ex-post returns because it will not include the returns on the day of the report after the release (unless the report was after hours) and often not for several days after dissemination. When we calculated using the last weekly price before the report date, the overall shape of the relative returns didn’t look that much different (ie: higher “red flags” was associated with lower relative returns), especially when looking out several quarters.

Click to see R table code

# Make datatable object
dt <-
  DT::datatable(
    annual_data[, {
      coverage = .N
      matched = .SD[!is.na(rel_ret_q_1), .N]
      percent_matched = matched / coverage
      list(coverage, matched, percent_matched)
    },
    fiscal_year][order(fiscal_year)], 
    rownames = FALSE,
    colnames =
      c("Fiscal Year",
        "New Constructs Coverage",
        "Matched with Returns",
        "Percent Matched"),
    options =
      list(pageLength = 24,
           scrollY = "400px",
           dom = 't')) %>%
  DT::formatPercentage(
    columns = 4,
    digits = 1) %>%
  DT::formatRound(
    columns = c(2:3),
    mark = ",",
    digits = 0)

Figure 2: Rate of New Constructs Covered Companies Matched with Returns Lower in Earlier Period

Figure 2 above shows the percentage of companies matched with returns over time, similar to Figure 1 earlier. We were able to download and match returns for between 75-80% of the stocks covered by New Constructs in the earlier periods, and a much higher rate in the later years. In the end, we have return data to go along with filings for almost 5,000 distinct companies, but were unable to match approximately 8,400 of the 67,000 annual reports for 913 companies. While we do have all the needed return data for many 2020 filings, the absolute number is lower because of the need to look ahead to calculate returns. We had significantly greater success matching quarterly reports, because those only started in a later period (around 2012 when New Constructs began providing them) when we had more complete pricing data.

Click to see R plot code

# Select cols with relative return data
cols <- 
    names(annual_data)[re2::re2_detect(names(annual_data), "rel_ret")]

# Melt on relative return amount columns
annual_data_long <-
  annual_data[, data.table::melt(
    .SD,
    measure.vars = cols,
    value.name = "rel_ret_amt",
    variable.name = "rel_ret_pd",
    na.rm = TRUE,
    variable.factor = FALSE,
    value.factor = FALSE
  )]

# Make ggplot using only 6 quarter subsequent returns and calculate median 
# by red flag
p <- annual_data_long[
  data.table::between(fiscal_year, 1999, 2020) &
    rel_ret_pd == "rel_ret_q_6",
  .(
    cases = .N,
    unique_companies = length(unique(ticker)),
    median_rel_return = sapply(.SD, median, na.rm = TRUE),
    mean_rel_return = sapply(.SD, mean, na.rm = TRUE)
  ),
  .SDcols = "rel_ret_amt",
  .(fiscal_year, total_flags)][, 
  ][, median_rel_return :=
        data.table::fifelse(
          median_rel_return < -0.5, -0.5, median_rel_return)][
  ][, median_rel_return :=
      data.table::fifelse(
        median_rel_return > 0.5, 0.5, median_rel_return)][
  ][,
    ggplot2::ggplot(
      .SD,
      ggplot2::aes(
        x = fiscal_year,
        y = median_rel_return,
        group = factor(total_flags),
        color = factor(total_flags),
        text = paste0(
          "</br>Reporting Period: ",
          fiscal_year,
          "</br>Total Flags: ",
          format(total_flags, big.mark = ","),
          "</br>Unique Companies: ",
          format(unique_companies, big.mark = ","),
          "</br>Cases: ",
          format(cases, big.mark = ","),
          "</br>Median Relative: ",
          scales::percent(median_rel_return, accuracy = 0.1),
          "</br>Mean Relative: ",
          scales::percent(mean_rel_return, accuracy = 0.1)
        )
      )
    ) +
      ggplot2::geom_line() +
      ggplot2::geom_point(size = 1)+
      ggplot2::scale_x_continuous(
        "Fiscal Year", 
        breaks = seq(2000, 2020, 5)) +
      ggplot2::scale_y_continuous(
        "Median Percent Change", 
        labels = scales::percent) +
      # ggplot2::scale_color_manual(labels = as.character(c(1:9)), values = c(1:9)) +
      ggplot2::labs(
        title = "",
        color = 'Num.\nFlags',
        caption = "Source: New Constructs") +
      ggplot2::theme_bw() +
      ggplot2::theme(
        plot.title = ggplot2::element_text(
          size = 10,
          face = "italic",
          color = "darkgray")
        )]

# Render as plotly with tooltips set in ggplot object
p <- 
  plotly::ggplotly(p, tooltip = c("text")) %>% 
  plotly::layout(
    hoverlabel = list(align = "left"),
    annotations =
      list(
        x = 1.05,
        y = -0.10,
        text = "Source: New Constructs",
        showarrow = F,
        xref = 'paper',
        yref = 'paper',
        xanchor = 'right',
        yanchor = 'auto',
        xshift = 0,
        yshift = 0,
        font = list(
          size = 12,
          color = "darkgray")
      )
  )

Figure 3: Earliest Period Had Most Red Flags and Differentiation among Performances by Red Flag

Figure 3 above shows the median returns for all companies over the subsequent 6 quarters (after reporting their 10-K filing). Using the tooltip, it is possible to see the number of companies, reports and flags during the selected period on the line chart. Note that we truncated the cases where the median return was worse than -50% in order to keep the scaling of the chart. There are usually only a few companies in these cases, but look at the “Mean Relative Return” in the tooltip these cases to get a better idea of the true number. The graphic shown here is static and only shows annual reports, but we will discuss how to change the parameters of the inputs using our interactive Red Flag Explorer Shiny app further down. In the next section, we will also discuss some of the ways that this picture probably understates the true differences in the relative performance of the “red flag” groups shown in the chart.

Bias From Missing Return Data

All in all, we collected almost 4 million weekly prices for 5,000 unique companies, and calculated the comparative log returns relative to the Vanguard Total Market Index fund (“TMI”) over the subsequent 1 through 13 quarters after every filing date. As shown in Figure 2 above, we were able to match several thousand stocks covered by New Constructs in most years. We expect that across all the periods and companies, the aggregated relative returns by “red flag” group (as shown in the “Summary Returns” table and “Returns over Time” chart on Red Flag Explorer), should be representative, though not precise, picture of the true returns for companies with those attributes. The bands for higher “red flag” groups will always have greater uncertainty, because there are a lot fewer companies as the number of flags increase.

The group of companies unmatched to returns is especially unfortunate, because, as shown in Figure 1, aggressive accounting and other risky behaviors appear to have been most common between 2000-2005, and a much larger number of reports had to be amended. Briefly in 1999-2000, earnings distortion occurred in over 30% of companies. It is likely that companies which went out of business (ceasing to generate pricing data), also would have had more warning signs, and thus, have been most relevant to our analysis. In fact, in a list of 38 the most notorious accounting- and/or fraud-related scandals we could think of (ie: Enron, Worldcom, etc.), our pricing sources were missing for 10, a much higher rate missing than the overall average. To further confirm this expectation, the average number of “red flags” in unmatched companies in the annual data and quarterly data was 1.7 and 1.5, respectively, compared to 1.4 and 1.2 in the matched group. As a result, we expect “survivor bias” to make our estimated returns for higher “red flag” groups somewhat less bad than they otherwise would have been.

Using the Shiny App

The triple horizontal line icon at the top left of Red Flag Explorer, opens a menu which allows the user to change any of the input fields. Changing inputs alters the data viewed for all of the tables and charts displayed when the app loads. The “View” switch allows to toggle between 67,000 annual and 78,500 quarterly filings. All annual filings are included as the fourth quarter in the quarterly data set, so remember that about 20,000 filings are used in both the series. It should also be noted that the same flag on the same filing, for a company and period, might differ between our annual and quarterly calculations. This is because flags are calculated with reference to the other filings of the same group (ie: annual or quarterly). It is possible to filter by sector or by selecting a customized group of companies.

When the app loads, it reflects the aggregate of all 13 quarterly relative return periods, but most of these periods encompass each other (ie: the one quarter return is part of the two quarter return, etc.). We have observed that it isn’t as easy to see the contrast of flag groups when the quarterly returns are only a quarter or two, so we recommend 4- or 6-quarters, but were unable to hard code this as a starting point into the app menu. The quarterly data set does not show returns beyond eight quarters, and though the annual data set shows up to 13 quarters, using this many will result in the loss of data in 2020 and 2021 (because we have to leave more lead time to calculate in those cases).

The “Returns over Time” tab shows the evolution of returns by number of red flags over time, similar to what we showed in the static Figure 3 above. There are generally only a small number of companies in the groups with such large declines. It is also possible to choose a selection of companies, but “Returns over Time” for a group loses its meaning if there are too few items, because those companies are always changing. If the selection is small enough, there might not even be a single data point for a higher number of flags in a given year, and as a result, there may be an extended time difference between two points when there were no members of that group between those periods (usually more than five flags in a period).

The “Flags Counts” tab on the app shows the descriptions and counts of all “red flags” included in the filtered selection shown on the plots. As mentioned earlier, most individual flags occur in approximately 20% of the filings, so seeing more than a few flags for any given company during a period tends to be the small minority of filings. The number of unique companies included in the “red flag” group is a more important indicator than the number of reports shown in “Returns over Time”. When looking at just one return period, this number should be pretty close to the number of reports unless an unusually long quarterly return period is selected.

Lastly, the “Counts over Time” tab shows the percentage of filings with that “red flag” during that period (as in the static Figure 1 earlier). Because “red flags” are calculated relative to the whole 20-year period, they are not evenly distributed by year to offer an objective perspective on how behavior in a given period and evolution over time. As discussed above, the early period in the chart has by far the most red flags, and in recent times, most groups haven’t come close to matching those levels.

Cursory Analysis

The beauty of our Shiny app is that we are able leave it for the user to explore the inputs and test sensitivity of “red flags” to performance for themselves. In aggregate, considering all periods, we observe that median group relative returns appear to be more stable and higher when the number of “red flags” is below 2-3, but generally seem to fall off proportionally as flags rise. There are naturally a lot fewer filings in higher flag groups, and as mentioned earlier, aggregate relative returns are much more uncertain and become considerably more volatile with a higher number of flags.

In the first few periods, most annual red flag groups outperform the TMI, but after 2003-04, almost all seem to under-perform, which we struggle to explain. This is even more so with the quarterly data, which starts later than the annual data. The TMI has a lot of stocks in it and should be less dominated by high market capitalization companies than the S&P 500. It also has a small management fee which our groups don’t have. We would expect the stocks doing better and worse than the index in any period, to cancel each other out in aggregate. If anything, it seems like our process for collecting returns would be biased towards companies which did well (by surviving), so we would have expected our average returns to do better than the TMI if anything.

In the year 1999 (the beginning of our data) and for a few years after, there was a much more significant number of companies with 5-7 “red flags”, and their under-performance was striking relative to the low flag groups. There are five years (2002, 2008, 2011, 2015 and 2020) where the differences among groups seems to contract, higher outperform lower red flag groups, and most groups outperform the TMI. These periods are all likely periods of changing expectations of Federal Reserve support for markets, and not surprisingly, this effect is by far the largest in the most recent intervention (during the 2020 Covid-19 period). Considering the whole period, it seems like the difference in median and mean relative returns between higher and lower flag groups seems to become smaller after 2005. We recall the post technology bubble period as being a golden age of stock picking. Certainly, when Mr. Gulliver wrote his Revelations in 2003, the case for indexing wasn’t as strong as it has become since. Maybe more market observers took note than we might have thought and arbitraged some of these effects away.

The separation among groups by number of flags tends to be persistent for most sectors, though seems a little more contrast in cyclical sectors. This may be because the several of the cyclical “red flags” (such as Margins & ROIC and Days of Inventory & Receivables) affect those sectors more and may allow for greater contrast. Some “red flag” groups which were prominent early on, such as “Earnings Distortion” and “Amendments”, mostly trended lower, while the prevalence of “Asset Turnover” warnings increased steadily over the period. Margins & ROIC declines rose sharply to above 30% during the 2008 and 2020 periods.

Thoughts about Data and Modeling

The model proposed by Revelations was rules-based, aggregating a large number of weak true/false (boolean) signals. It’s power comes from simplifying and systematically taking more elements over a larger number of companies than most single analysts might consider at the same time. By collapsing the financial ratios down into boolean using arbitrarily thresholds, we lose a lot of data and are unable to allow variables to interact. We are also treating all of “red flags” equally, when some may be more important on their own or in combination with others. New Constructs was kind enough to share their data, but that was on a temporary basis, so this is beyond what we can achieve in this project, but taking these into account might give a even better signal of future performance.

In addition to the problem with sparsity among the higher flag groups, it isn’t clear if the relationship between the number of flags and relative returns should be linear, and the few very filings with a high number of flags often seem to have had a more than a linear impact. The statistical modeling tools we have learned so far rely primarily on a few highly significant numeric variables, so we are not sure how to model when the signal derives from the sum of many weak true/false variables.

The nature of the data is also a consideration. We would like to measure the true difference in relative returns of an individual or collection of flags. The challenge is the longitudinal nature of the data, and the related sparsity of high flag data by period. This is a situation may be best addressed with a Bayesian mixed-effects model, which might allow us to get a more confident estimate for the return differentials of the sparse flag groups, by taking into account measurements from other periods to reduce uncertainty. This is something we may attempt in the future.

Next Steps

There are so many possible next steps it is difficult to know where to start. The first place would be to find the price histories for the missing 900 companies, preferably adjusted closing prices for comparability. It also might make sense to conduct as a genuine back-test of returns at different “red flag” levels, but we would need to re-format our rolling join to make the purchase after the report, and our data wouldn’t allow for this consistently on the same date. We have included mainly accounting ratios in our analysis so far, but we could also easily add variables from other data sources, such as NLP signals like the sentiment of the MD&A, management incentive alignment measures, short interest ratios or insider selling. At the moment out of almost 145,000 filings, we found only about 2.5% with five or more “red flags” among the annual and quarterly data, so the contrast among groups might increase further if we to took into consideration more risk factors. With our code in an R {targets} project, any of these additions would only a short time to add to the app, so the whole project becomes a living document.

In our next blog post in this series, we plan to add a series of filters to the app of several themes including: notorious accounting collapses, meme stocks, high internally-generated intangible, companies which were present over the whole period and companies with high returns in spite of high red flags. We might also add some more visualizations to the app to enable users to explore these filters. Finally, we may attempt to build a Bayesian mixed-effects model in a future post as described above in order to better quantify if the differences and evolution in relative returns among groups we think we see in the “Returns over Time” plot are in fact real.

Conclusion

When we wrote A Blueprint of “Red Flag” alerts Using Adjusted Earnings Data, it felt like our goal was at the least a stretch. Thanks to the generosity of New Constructs and the powers of R, we have taken a long-held research question, and in our Shiny app, built a working prototype, which allows our findings to be shared with others and added to over time. The nature of the project is to see how effective the strategy is at filtering the small group of companies likely to have poor returns. Because of the sparsity of high “red flag” companies (which is by design), we struggle to know how to prove that the effects we think we see in the charts represent true differences in performance or are changing over time. Though we also cannot prove it, we suspect that the adjustments New Constructs has performed are likely a vital part of the model’s ability to separate stocks into performance groups.

We plan to include the code for our calculations and Shiny app, and the derived “red flag” data on Github. As a final note, we conducted most of our research on our own, and though we have been careful and re-checked many times, our code and calculations has not been reviewed by others. It is possible that there are errors in our analysis, and this work should not in any way be considered investment advice. Our work is only a back-test, so we encourage others to consider subscribing to New Constructs API and their real time data to mine for possible future accounting troubles.

Introducing the Redwall IRS SOI Tax Dashboard

Fri, 06 Aug 2021 00:00:00 +0000

Introduction

The IRS SOI Tax Statistics is a fabulous seemingly undiscovered data set, which should be frequently referenced in any discussion about income and taxes. We often see median income represented by Census data, but have always been skeptical that most people know their exact income in any given year or would report it according to a uniform definition if asked by a census taker. This data set aggregates elements of all 1040 filings by zip code across six income levels, with Adjusted Gross Income (AGI), sources of income, credits and deductions with both amounts and counts. It seems to reflect a much more robust measure of earnings, although a slightly less granular format than the census tracts. As is typical in almost every open public data we have worked with, formats, variables and groupings change from year to year, so accessing a clean long-term time series was a painful proposition.

We first introduced the IRS SOI Tax Statistics as a thematic data set for Redwall in Analysis of Connecticut Tax Load by Income Bracket. In that post, we discussed the data and methodology used to collect and clean it in. Then, we used it in more detail in IRS Data Shows Growth in Number not Income of Highest Earners since 2005. Finally, IRS Data shows Connecticut Taxpayers Also Pay Higher Federal Taxes compared the federal tax rates payed by CT residents to national averages. All three of these previous posts were before we had the disclosure for 2018, one of the more interesting tax years because of the Trump tax cut and its controversial SALT limitations.

When we worked on those posts, we also did not have the skill we have now have to make a Shiny dashboard, unlocking this valuable data for anyone wishing to explore it. In this post, our we discuss our new {irs.soi} package (available on Github, not CRAN), describe the methodology we created to download, clean and prepare the data for exploration in a new Shiny {golem} app.

Methodology

One of our themes in recent months has been re-factoring the messy code and work flows, that we built up while learning over these last few years. Like many, our IRS explorations had become a mess and certainly not accessible to anyone else, so moving all of our functions into a package structure was overdue. Hence, we built the {irs.soi} package, now hosted on Github. We found the NBER’s Public Use Data Archive cleaned up version easier to use than the daunting collection of .csv’s on the IRS website, so the package downloads specified years from the Individual Income Tax Statistics - ZIP Code Data (SOI) to a specified local folder. Please note that our app data includes 2017-2018, previously included on the NBER site, but now removed. Using {irs.soi}, specifying a path and running download_nber_data() will download the data to a local folder (or put it in a data/ file if none is specified). Please note that the full data set is over 3GB and should take at least 15 minutes to download.

Again using the {irs.soi}, the load_soi() function can be used to load and compile the data into a data.table, and clean_soi() cleans according to our process. One place we had to manually intervene was to aggregate income levels, because the IRS had changed size bands in 2007. We did this in a way which maintained the integrity of the bands (ie: <$10k and <$25k became one band and >$100k and >$250k another), but at the expense of interesting granularity. Over the last 10 years, >$250k has been generally been defined as “rich”, but our highest bands starts well below that. There were also some cases in 2006 where the data had bad classifications, but we were able to impute, and 2007-2008 strangely did not round by 1,000 as the other years did. Our prepare_app_data() filters sparse zip codes, and merges by zip code to add County and City filters, and can be used to ready for it to be run locally in Shiny using our irsApp() function.

Data Summary

To give a quick summary of the raw data and our preparations. There are almost 42,000 unique zip codes in the US, but many of these are P.O. Boxes with few or no people, and others are suppressed from disclosure if they don’t have many taxpayers in a given period. We also further screened in {irs.soi} for only zip codes where all income levels were included for every year of our specified data, which gave us a total of 24,143. The IRS data is only disclosed by zip code and state, but we added County and Post Office City fields by joining with the zip code database in {zipcodeR}. The IRS removes zip codes below a threshold of filers for privacy reasons, and our package drops all zip codes which are not present in all years of the data set, which cost between $100-150 billion of annual income in most years, but our final data is still close the full amount of returns, income and tax paid as reported by the Tax Foundation. The Tax Foundation reported total US taxpayers of 143.3 million, earnings of $10.9 trillion and taxes of $1.6 trillion in 2017, so the app pretty much matches these numbers in aggregate, but offers the opportunity to drill down by state, county, city and zip code, to see income, deductions and taxes broken out by major sources, as well as seeing an additional 13 years of data. Perhaps the most interesting aspect of the data in the app is that it includes 2018, the first year including the Tax Cuts and Jobs Act of 2017.

Conditional Drop Down Filtering

This is not a code-based post, but just a word about the filtering of locations, which is now excellent. The app has a nested geographical structure. We wanted users to be able to find their chosen geography with as little struggle, waiting time and server usage as possible. We tried several ways of doing this efficiently, but it was difficult. In Hadley Wickham’s new book section, Mastering Shiny: 10.1.2 Hierarchical Select Boxes proposed a sequence of nested observeEvent() and updateSelectInput() operations to filter a reactive data.frame inside the server function. We also looked at doing this on the server side with updateSelectizeInput, but with the size of our data or inexperience, it was very slow and usually ran out of memory. For others looking to accomplish a similar task, please see our code using shinyWidgets and its selectizeGroupUI and selectizeGroupServer. This solution may be widely known to Shiny experts, but little has been written about it. The credit for helping us goes fully to David Solito for Filtering a data frame with dependent drop down lists in Shiny thanks to Magrittr (aka conditional drop down list).. This was an excellent summary of a complicated task, which seems to be by far the best solution for this seemingly common task.

Using the Shiny App

We were going to include the app in this blog post, but it is large and may take a little time to load, so please read the post and then follow the links to explore. When the app launches, it displays the aggregate AGI, tax paid, returns and unique zips for all of the data, included on the “Summary” table tab as shown at the beginning of the post. Choosing the “Income” tab, key component income sources, such as Salary, Dividends & Interest, Capital Gains, Business and Partnership Income and Pension & Social Security Income are displayed, can be explored over time.

IRS Dash Income Tab

By choosing the "Deductions tab, Total, State Taxes, Local Tax, Mortgage and Charitable Deductions in aggregate over time is displayed.

IRS Deductions Tab

Using the three horizontal line icon at the top left, it is possible to access other tabs and filters. There is a second tab called “Charts” to see a plots of the change in AGI and tax rates over time. It is possible to see Aggregate or Per Capita by selecting with the “View” switch. The second tab also includes the “Compare” tab displaying the evolution of every state over time. Exact amounts or counts can be viewed by hovering with the tooltips.

IRS Dash Per Capita AGI over Time

As mentioned above, it is possible to filter all of these tables and charts by State, Counties, Cities or all the way down to individual or groups of zip codes using the black tab on the left. There is also the option to select income bands ($<25k, $25-50k, $50-75, $75-100k and $\100+). We would have liked to show more granularity above $100k, but that would have meant showing a shorter time period. Given the richness of this data set and the incredible functionality of Shiny and the flexibility of {golem}, we hope to add many more avenues of exploration in the future.

Assessing the App

At about 110MB and almost 1.7 million rows, this app is large, and may have a broader audience than some of our previous ones (Connecticut Property Selling Prices vs Assessment Values over Three Revaluation Cycles and Risk Score History of Selected CT Town, so we are not sure how long our server time on Shinyapps.io will last at our account level and advise patience if it takes a moment to load. We intend to add more fields and visual analyses, and to expand the functionality, so have to explore the options for cost effective hosting, since this is personal project. If the app runs out of server time, there is always the option to use {irs.soi} to download the and clean the data, and then run the app locally with {irsApp} (as described above).

Observations

We observe that income of lowest income group viewed on national level has increased average AGI by about 20% since 2005, and there are significantly fewer filers in that group over the period. Average income levels of the three middle bands are so flat in nominal terms over the period that it is almost hard to believe. There were increases in the number of tax payers in all of the middle groups, but not even close to increase in number of taxpayers in the highest income group, which more than doubled over the period to 28.6 million returns (out of 146 million filed in 2018). Unlike the other groups, the average income of the highest group declined considerably during the Great Recession, but has more recently recovered to be slightly higher than the previous peak. In real terms, a significant loss of purchasing power on average in all groups over the period is evident. If greater granularity in the highest group were possible, it might show the much discussed gains made by the top 0.1%.

In aggregate, total deductions fell from $1.3 trillion to $600 billion between 2017 and 2018, with big declines in state, local, mortgage and charitable deductions. We will leave it to explorers to find out which geographies suffered the most by exploring using the filters. in spite of the lost deductions, the effects of the recent tax cuts can be seen in all income bands having lower Federal tax rates in 2018, and even the highest bands of most blue states show lower tax rates despite SALT restrictions. The highest income bands did indeed see a reduction in percentage of total tax paid during 2018, to 79.5% from 81.7%, though in our bands, this includes incomes could be considered middle income in high cost areas.

Conclusion

We have probably reached the size we can with our current shinyapps.io infrastructure. We plan to add more visualizations and tables with the existing data, but would also like to add additional variables from other sources, such as real estate prices, transportation or education data in the future. In order to do this, we will need to learn a considerable amount about how to deploy on Shiny Server or create our own server on AWS in a Docker container. The code to download the IRS SOI data and the app are available on Github for others to try. We would welcome other partners, contributors or just feedback to improve the app in the future.

A Blueprint of "Red Flag" alerts Using Adjusted Earnings Data

Wed, 21 Apr 2021 00:00:00 +0000

Photo by Valenciana Nicolau Primitiu via Unsplash

Introduction

In A Walk Though of Accessing Financial Statements with XBRL in R - Part 1 and Learning SQL and Exploring XBRL with secdatabase.com - Part 1, we set out with the hope of building a model to mine for “red flags” in financial statements following a roadmap set out in by Bruce Gulliver in the 2003 AIMR Conference Proceedings Revelations in Financial Reporting. Unfortunately, this paper is still behind a pay wall, but it is one of the most valuable research publications we have seen from the CFA Institute. We wish it were in the public domain after almost 20 years. Mr Gulliver used a similar methodology at his firm Jefferson Research to generate “Torpedo Alerts”. We don’t know the status of his work more recently, but he could not be reached and the Jefferson Research website appears to no longer to be live. Our thanks for the way he synthesized and shared his research in a simple and understandable framework, and all the credit for the blueprint we are about to lay out here, goes to him.

While both of the previous posts mentioned above started out with ambitious intentions, neither could be completed with the available machine readable financial statement data. As reported, aggregated raw XBRL of many companies and periods didn’t allow for meaningful comparisons even with a company’s own past reports, because of changing tags over time. Comparisons with close competitors and those in other industries, are also complicated because of the leeway which companies have to customize reporting tags under XBRL rules. Finally, raw reported XBRL doesn’t offer ready tools to adjust the income statement for items in the MD&A and footnotes in the way we would like. We were probably naive to hope for this, though we can still wish for a full open data set of the clean quarterly financial statements of all public companies, and all the adjustments for upload into analytic software environment like Python or R with the click of a mouse (like we have been able to do with so many other important open source data sets).

Though contrary to our goal to stick with mainly open source data in this blog, we concluded that we may have to explore commercial sources. We recently learned that New Constructs (NC) has built a perfect data set for this purpose, which we will discuss in the next section. We were able to access New Constructs’ website to explore stock-by-stock data within their UI, but not to mine the full set of data as a whole in R. Please do not read on if expecting to see a completed study or code in R, but our intention is to pass on our findings and lay out a blueprint of what long-term forensic accounting “red flag” analysis might look like for others in the future.

New Constructs

Last month, we learned of NC from their joint paper Distribution Is Not Enough. Having operated since 2003, the accounting research firm had gradually built an impressive process combining man and machine, which has enabled them to scale up and reduce errors when conducting financial analysis on so many companies. This data set is in a different league from what we had seen with SEC Edgar, Financial Modeling Prep or SECDatabase.com. NC has gone through every earnings report since the late 1990s and adjusted annual 10-K’s for items left off the face of the balance sheets and income statements, but included in the MD&A, footnotes and cash flow statements. They also have similarly adjusted quarterly data since 2013. We would advise anybody looking to learn about this topic to read their Learn From Value-Investing Experts, and particularly the section about “Accounting Fixes”, it is like a textbook on its own. They also offer many excellent webinars on the subject.

A team of Harvard and MIT researchers, who had access to NC’s data, recently published Core Earnings: New Data and Evidence in the Journal of Financial Economics verifying the accuracy in a sample of reports and comparing the overall quality to Compustat. Though there are armies of analysts pouring over financial statements, they also demonstrating that NC’s adjustments were not really discounted by the market (finding an 8% excess return in back tests when comparing the companies with the most to the least adjustments). By the researchers count, NC found that approximately 30% of non-core items or almost 20% of net earnings over the period where non-core items dispersed throughout the 10-K, on and off the face of the income statement. They estimated that roughly half of these items on average were off the income statement (certainly a material amount), and found that the quantity and magnitude of such items had been increasing over time. The researchers found that NC’s core earnings had a significantly higher year-to-year persistence (ie: future earnings were more highly correlated with past periods once non-recurring items had been removed), and hence represent a better baseline in any model.

Mr. Gulliver never outlined how he cleaned his data although he did mention a service called Simplystocks, which was purchased by S&P Global Intelligence in 2003. The main comparison used in the Harvard/MIT paper were with Compustat. In this post, we will outline the model we built as if we would be able to back test Mr. Gulliver’s “red flags” (using NC’s superior data). We found that the NC’s variables as shown in their Data Feeds & Dictionaries. Without the access to the data, we thought of dropping the post, but having spent the time to do the research, we wanted to share the ideas and information with others who might have greater resources than ours. This post will be a walk-through of what a process like this would look like in hopes we one day get access to similar data.

The Forgotten Art of Forensic Accounting

In the best of circumstances, it has been challenging predicting winners and losers with the rise of the digital economy using reported GAAP metrics. Still, we believe that financial reporting remains a vital, although incomplete component of an investor’s toolkit. Just to cite some of the sources that we used to prepare for this project (aside from “Revelations in Financial Reporting”). Mr. Gulliver himself recommended Quality of Earnings by Thornton O’Glove (1987) and Financial Warnings: Detecting Earnings Surprises, Avoiding Business Troubles and Implementing Corrective Strategies by Charles W. Mulford and Eugene E. Comiskey (1996). We also purchased Mulford & Comiskey’s The Financial Numbers Game: Detecting Creative Accounting Practices (2002) in our research.

The most recent of these sources is almost 20 years old at a time when arguably it should be most relevant (more than a decade into an historic bull market). When we look at it in this fantastic new tool called Connected Papers (graphic above), we see that Mulford’s 2002 book stands completely on its own with no connections and among other ancient papers. We could speculate about why this approach to investing feels so out of favor when data mining would seem to be at the top of mind. In the current environment, as seen after other extended bull markets, there is the feeling that people are more willing to invest in concepts and with less concern for future cash generation. But, this trend has been increasing for more than two decades, even prior to the GFC. It could be a bubble, but it also correspond with the shift of firm’s investment spending from fixed and measurable to internally-generated intangible assets as Sparkline Capital showed in the graphic below from their recent Investing in the Intangible Economy). In a future post, we will explore this possibility at greater length.

We would have guessed that the rise of free analytic software and machine readable data like XBRL would have led to a revolution in financial statement sleuthing, but we were wrong. Core Earnings: New Data and Evidence is also a lonely paper, although there are a few more recent papers which are similar.

Key Accounting Diseases

For this post, we will stick to the goal of possibility of mining the full financial statement corpus for “red flags”. Mr. Gulliver gives four categories for diagnosing what he calls the “diseases”:

Cash Flow Quality
Earnings Quality
Efficiency of Operations
Balance Sheet Quality (Liquidity)

In his “Torpedo Alerts” research, he also added a final screen for valuation. Within these larger categories, he often offers several sub-category, which we outline below. We will choose variables from NC’s website which we thought would best replicate these filters. The other key consideration is how to calculate and weight the “red flags”, because some must be more significant than others. As far as we can tell, Mr. Gulliver used equal-weighting, but with the data loaded up in an analytic software tool like R and Python, we would be able to iterate and even let a model tell us the most meaningful coefficients in predicting earnings warnings and/or significant stock price adjustment.

We would also be able to generate more complicated variables. Mr. Gulliver appeared to mainly use single year-on-year changes, but we would try out increasing the weight of persistent negative year-over-year changes over several years. Also, Mr. Gulliver did his analysis using quarterly data, but NC’s has only annual data before 2013, so we would have to mine for bigger disappointments if we wanted to use the longer time periods.

CASH FLOW Quality

Mr. Gulliver argues that an increasing gap between reported and adjusted cash flow is an indicator of poor cash flow quality. We would compare the trend in reported Operating Cash Flow with NC’s Free Cash Flow giving a penalty for the difference as it grows sequentially over reporting periods. His Torpedo methodology also compares Operating Cash Flow to Current Liabilities. We had not seen that as a significant factor, so perhaps we would consider giving this a lower weighting in the overall measure.

Torpedo	NC Ticker	Comment
Op/Free Cash Flow	CASH_FLOW_OPERATING	Supporting CF - OpCF/FCF
Op/Free Cash Flow	FREE_CASH_FLOW	NC version of FCF
Flow Ratio	CASH_FLOW_OPERATING	Operating CF/Current Liab.
Flow Ratio	LIABILITIES_CURRENT

EARNINGS QUALITY

A/R & Inventory (Balance Sheet)

The change in Inventory or Receivable levels relative to revenue run rates can be early indicators of changing interest in a company’s product or services. We propose to use a 3-year weighted average, with a growing penalty if the trend in Days continues to deteriorate year-on-year for more than one period.

Torpedo	NC Ticker	Comment
Inventory	DAYS_IN_INVENTORY	Weighted change
Receivables	DAYS_IN_REVENUE_OUTSTANDING	Weighted change

Accruals (Balance Sheet)

Similar to Inventory and Sales Days Outstanding, the overall change in Reserves per change in revenue could be an early indicator of a change in operating conditions as a company reaches to meet expectations with less conservative assumptions. The NC RESERVES variable includes the ending balances for the LIFO, inventory and accounts receivable reserves. Reductions in these accounts relative to the underlying rate of growth in the business may indicate that the company is postponing current expenses for future periods, gradually making expectations more challenging to achieve. In a single year or on their own, such a movement might not say anything, but combined with other signals, may be significant.

Torpedo	NC Ticker	Comment
Accruals/Sales	RESERVES_YOY_DELTA_PER_REVENUE	Reserves/Revenues
Accruals vs. Sales	None	Unsure what of diff with Accruals/Sales
None	RESERVES_PERCENT_IMPACT	Reserves divided by Net Assets

Earnings Quality (Earnings Quality)

NC takes pains to adjust earnings to remove items in the MD&A and footnotes which are non-recurring (positive and negative), but have been included in the main financial statements. NC also adds items which affect earnings, which have been excluded in order to derive its CORE_EARNINGS_AFTER_TAX. It then derives EARNINGS_DISTORTION based on the difference between the derived Core Earnings and the GAAP Net Earnings. We would use the EARNINGS_DISTORTION as a percentage of net income and net assets as indicators of earnings quality, and likely a superior proxy to Mr. Gulliver’s measurements. Thorough adjustments such as these are the main challenge, and we doubt there would be a more thorough and comprehensive data set for this purpose than that of NC.

Torpedo	NC Ticker	Comment
Net Income	None	Skip
Adj Income	None	Skip
Core Income	None	Skip
None	EARNINGS_DISTORTION_TOTAL_PER_ASSETS	Divides Net Inc by Core Inc
None	EARNINGS_DISTORTION_TOTAL_PER_INCOME_NET	Divides Net Inc by Core Inc

EFFICIENCY

Returns (Efficiency)

Mr. Gulliver used ROE and ROA in addition to ROIC and CF ROI as measures of returns, but these are again derived from reported GAAP net income. NC makes the case against using reported ROE in Don’t Get Misled about Return on Equity (ROE)](https://www.newconstructs.com/dont-get-misled-by-return-on-equity-roe/), so we would rely on NC’s metrics (as built into its ROIC and Invested Capital). NC charges companies for past write-offs, by adding back to Shareholder’s Equity (so management has to carry the full weight of past missteps and can’t hide behind “serial charges”). In the case of NC’s RATING_ROIC, returns relative to Invested Capital are bundled into five categories from relatively low to high. According to NC’s methodology, ECONOMIC_PROFIT is the amount of ROIC in excess of WACC times Invested Capital as a hurdle rate or charge for use of the capital during the period. As for Free Cash Flow, NC explains their metric in Free Cash Flow And FCF Yield. For our purposes, we would use the Free Cash Flow per Invested Capital. Because NC doesn’t provide a Free Cash Flow Yield rating, we might construct an index for the companies in that industry during the period and cut them into five categories according to SIC (Industry) code. The beauty of working with R is that we can quickly iterate and reshape variables in any form we would like once we have the data in our environment.

Torpedo	NC Ticker	Comment
ROE/ROA	None	Skip
ROIC	RATING_ROIC	NC rating version grades ROIC level
None	RATING_INCOME_NET_ECONOMIC_PROFIT	Economic EPS vs Reported Ranking
CF ROI	FREE_CASH_FLOW_PER_CAPITAL_INVESTED	NC version measuring Cash Generation

Margins & Return Trends (Efficiency)

This category is distinguished from Returns above in that we would consider using the 3-year weighted trend with growing penalties for ongoing negative variance (instead of the absolute level of returns relative to peer companies). It is easy to understand that negative incremental changes in margins and returns on capital could be an early indication of deteriorating operating or business conditions. For example, declining gross margins might indicate a weakening competitive position, and higher overhead costs through increased SG&A could pressure EBIT margins. NOPAT might reflect increased tax rates, and ROIC that the company might be effectively buying more business. Since we are looking for combinations of factors across many metrics, we are able to spread our net widely to capture any possible signal of deteriorating business conditions for an enterprise and allow a seemingly small change in one margin might interact with another in the cash flow, balance sheet or earnings quality metrics.

Torpedo	NC Ticker	Comment
Gross Margin	PROFIT_GROSS_PER_REVENUE	Calculate weighted 3 yr Trend
EBIT Margin	EBIT	Calculate weighted 3 yr Trend
EBIT Margin	REVENUE
SG&A Margin	EXPENSES_SGA	SG&A / Revenue
SG&A Margin	REVENUE	Calculate weighted 3 yr Trend
None	RESEARCH_AND_DEVELOPMENT_EXPENSE	Calculate R&D / Revenue
None	REVENUE	Calculate weighted 3 yr Trend
NOPAT Margin	NOPAT	Calculate NOPAT /Revenue
NOPAT Margin	REVENUE	Calculate weighted 3 yr Trend
ROIC	ROIC	Calculate weighted 3 yr Trend
Tax Rate	Skip	NOPAT margin seems sufficient

Turnover (Efficiency)

Mr. Gulliver used the inventory and receivables 12-month turnover ratios as indicators of an efficiently run business, but we would use the full working capital efficiency given the available NC metric. This metric uses current assets minus non-interest bearing liabilities and is scrubbed of non-operating items. We might construct ranking groupings relative to other companies in the same industry (by SIC) as we did with the other efficiency metrics above. This is distinguished from the Inventory and Receivable days outstanding in the sense of being a one-year metric and with a ranking instead of a signal of change in operating conditions leading to inventory build-up or slow sell-through of products. He also uses reported Assets/Equity, so we would use NC’s fixed income EQUITY_MULTIPLIER, which is a reflection of Adjusted Assets over operating Shareholder’s Equity.

Torpedo	NC Ticker	Comment
Asset/Equity	Unknown	Fixed income Equity Multiplier
Receivables 12 mo	CAPITAL_WORKING_NET_TURNS	Combine
Inventory 12 mo	CAPITAL_WORKING_NET_TURNS	Combine
None	CASH_CONVERSION_CYCLE
None	CAPITAL_WORKING_NET_YOY_DELTA_PER_REVENUE_YOY_DELTA

LIQUIDITY

Liquidity (Fixed income data set)

After all that work to clean up the financial reports, why not also construct fixed income ratings, which is exactly what NC did. Mr. Gulliver uses current ratio and quick ratio, so we would use the same as they are already available in the NC fixed income database and likely not needing adjustment. NC itself uses the 3-yr average FCF-to-Debt and EBITDA-to-Debt as its liquidity proxies, so we would use the available DEBT_NET_OF_CASH_PER_EBITDA and calculate EBITDA-to-Debt using NC’s DEBT_FINANCING and the adjusted EBITDA. NC has a really interesting metric (EXCESS_CASH), which is the cash retained, but not needed to maintain operations. It seems like changes in this especially around zero might offer information. It is entirely possible that a company in a tight liquidity situation might offload inventory at lower prices to the detriment of margins or start to obtain less favorable terms from suppliers.

Torpedo	NC Ticker	Comment
Current Ratio	Unknown	Fixed Income data set
Quick Ratio	Unknown	Fixed Income data set
Cash	EXCESS_CASH	Calculate Debt - Excess Cash/3-yr avg FCF
None	DEBT_FINANCING	NC Total Debt
None	EBITDA	NC Adjusted EBITDA
None	DEBT_NET_OF_CASH_PER_EBITDA	NC Liquidity measure

VALUATION

Valuation

Valuation was not part of Mr. Gulliver’s CFA Revelations paper, but he understandably used one for work with his firm in order to make investment decisions. Obviously, a highly valued stock which missed expectations would have more downside. NC has spent more than a little time explaining why traditional metrics like P/E, P/Cash, P/Sales and PEG ratios are poor indicators of stock performance in Basic Metrics. Without going into great detail, NC found a more robust link between EV/IC and their adjusted ROIC than any of these other metrics, so we would use these.

Torpedo	NC Ticker	Comment
P/E	Skip	NC version seems better
P/Cash	Skip	NC version seems better
P/Sales	Skip	NC version seems better
P/Growth	Skip	NC version seems better
None	CAPITAL_INVESTED	EV/IC
None	ENTERPRISE_VALUE
None	RATING_GAP	Proxy for PEG
None	RATING_PRICE_TO_EBV_RATIO	Proxy for P/Book
None	RATING_FCF_YIELD	Proxy for P/Cash

OTHER ORTHOGONAL FACTORS

Revelations also did not discuss potential orthogonal factors directly, but a change in the Auditor, changes in significant accounting policies or a qualified opinion are fields in the NC database which might be risk factors. Frequent amendments are certainly a risk factor, but NC do not keep track of past filing changes. Natural language processing could be used to assess changes in tone by the management in the MD&A. Factors for executive compensation alignment might be added, and indeed NC does have a ranking for this in its data. Aggressive accounting might be more prevalent depending on industry. For the purposes of this exercise, we would need to remove cases where the filing has already been amended as the underlying problem would already have been discovered and no longer a risk factor. Other future addition would be to include the short interest ratio and insider selling metrics. Data such as these could be easily added to the analysis with R, so that is a future opportunity.

Torpedo	Ticker	Comment
None	Unknown	Auditor
None	Unknown	Auditor Opinion
None	Unknown	CEO name
None	Unknown	CFO name
None	Unknown	SIC code
None	Unknown	filing type (ie: 10-K, 10-K/A)
None	Unknown	Report date
None	Unknown	Filing date
None	None	Restatements
None	None	Management Incentives
None	None	Short Interest
None	None	SEC Insider Selling Metrics
None	None	Dividend coverage
None	None	Stock Price on reporting date

Conclusion

In the old days, we used Datastream and then Bloomberg to look for insights in financial data, so it is surprising that such a data rich field as financial analysis feels like it stalled out compared to the rich new world of advanced analytics. The main potential sponsors, like the SEC, CFA Institute and AICPA, don’t seem aware or to have fallen behind in adopting the habits of the free and open source data. The opportunity to look at all core earnings from all of the companies in one data set, and to iterate over it in search of revealing patterns, is an exciting prospect, and we have written several posts to try to get there. We read Mr. Gulliver’s piece in 2003 when there was no R or Python, no way to get data in that form, so building something like that was unimaginable. Now it feels like it could take a few weeks to build something really exceptional, tease out some real signals and maybe cause a management inclined to push the envelope to think twice. If any readers are aware of research or data available along these lines, we would appreciate knowing.

Using Census of Govts to Show that if You Know R, You Know SQL

Sat, 06 Mar 2021 00:00:00 +0000

The Government Finance Database Viewed in RStudio Connections Pane

Introduction

We have been exploring Willamette University Government Finance Database, a cleaned up and aggregated version of 50 years of annual Census of Governments. For those interested in learning more, please see Willamette’s paper The Government Finance Database: A Common Resource for Quantitative Research in Public Financial Analysis describing the database and its creation. It is an amazing resource of 2.4 GB in .csv form, but with over 500 fields of financial statement items in nested levels of government entities, is very complicated even after the extensive work Willamette have done to compile it. This is exactly the kind of less well known, but valuable longitudinal data, for which we have had the most interest. Our recent Introducing the Redwall IRS SOI Tax Dashboard describes our similar project with 15 years of detailed IRS tax data by zip code, and related new package {irsSOI} and {golem} Shiny app {irsApp}. We dug a bit into SQL in Learning SQL and Exploring XBRL with secdatabase.com - Part 1, but it took a long time to generate queries we felt confident in. In this post, we will describe how we can use the amazing {dplyr} SQL translation layer as a tool to quickly develop and explore with elaborate queries. This may also help us to find a durable solution of a back-end for our projects and apps with {SQLite} databases.

Where This is Going

When we began, it was fine to load a .csv file into memory, but as our projects, size of data have grown, and most urgently, the need for responsive apps backed by larger amounts of data, a strategy to use remote relational databases has become more pressing. We looked at solving this problem with Tapping Yelp data with Apache Drill from Mac using {sergeant}, but Drill was complicated to install, and ultimately couldn’t get it to work as hoped. With {SQLite} widely available and built into our local system, ready to store and fast retrieval from an RStudio connection with a simple line of {dplyr} code seems like a winner. Our ultimate goal find the right back-end for larger data, allowing fast, easy filtering and aggregations across many categories in Shiny apps. When we built the trial version of IRS Tax Dashboard, we found that the app was very responsive despite having tens of thousands of nested filtering options backed by a .fst file, but could only host a fraction of fields that we hoped to include on our introductory Shinyapps.io account. We now have an much reworked {golem} version ready to go with many more fields and viewing modules, but at 300MB+ of data, it is too big to deploy through that channel. With the {SQLite} solution, all that remains would remain is to set up to access the database from an AWS in a docker container.

The Willamette Government Finance Database and `{govFin}` Package

For this project, we built {govFin} to download and convert the Government Finance Database’s aggregated .csv into a {SQLite} database all with one function shown in the code chunk below. The downloaded zip file is 292 MB, but expands to into a 2.4 GB csv, before fitting in a 1.1 GB database. With separate tables for each type of government entity (federal, county, state, local, school districts and special districts). Response to queries lagged a bit until we added the Census identifier (“id”) as index for all the tables. After that, responses to joins and aggregations were instantaneous. We have set up separate tables for the identifier and population counts also indexed by “id”.

Within tables, there is still a lot of complexity, flattened with detail and summary rows included side-by-side for balance sheet, income and cash flow items. There are also two separate surveys, the full census every five years and annual surveys which have a lot fewer responses and are biased to larger municipalities. For those wanting to replicate some of the strategies in this post or to access the data, please install our {govFin} package from Github. Running govFin::full_gov_census_db_wrapper() with the desired local path will download the full Willamette data set and build a {SQLite} database leaving behind only the original zipped folder.

# Not run here
# install.packages("devtools")
devtools::install_github("luceydav/govFin")

# Download zip and create SQLite database at path
path <- "/Users/davidlucey/Desktop/David/Projects/govFin/inst/extdata/"
govFin::full_gov_census_db_wrapper(path = path)

The `{SQLite}` Database

Once built, the database can be easily accessed from RStudio with the DBI package, but unfortunately a SQLite connection in memory can’t be viewed in the Connections pane. It possible to use Connections (pictured at the beginning of this post) by using connections::connections_open() instead of DBI::dbConnect(). However, {connections} didn’t allow us to interact with the database using DBI functions like DBI::dbListTables() or DBI::dbListFields(), so we went with {DBI}, because we are going to use the {dplyr} tbl() function to create a table object in the Global Environment and to interact with tables anyhow.

# Set up connection with DBI package
gov_census <- 
  "/Users/davidlucey/Desktop/David/Projects/govFin/inst/extdata/gov_census.db"
conn <-
  DBI::dbConnect(RSQLite::SQLite(), dbname = gov_census)

# Alternative to use RStudio connections pane
# conn <-
#   connections::connection_open(RSQLite::SQLite(), dbname = gov_census)

As shown in the code chunk below, there are 526 fields in our local table, and when considering that it is relatively consistent over so many entities and years, is a impressive resource and surprisingly little cited and discussed.

Click to see code generating output

d <- DT::datatable(
  data.frame(
    fields = DBI::dbListFields(conn, "local")[1:175], 
    fields2 = DBI::dbListFields(conn, "local")[176:350],
    fields3 = DBI::dbListFields(conn, "local")[351:525]), 
  rownames = FALSE
)

There are large number of financial variables spread across a many tables, and also some identifying data stored in “id_cols” and time-related demographic data in “popu”. In order to aggregate values per population by state or local entities, complicated {SQL} queries involving five separate tables (“state”, “county”, “local”, “school” and “special”) are required. We would also need to separately link our id and demographic data which are also in separate tables. This post will show how easy this can be using a familiar {dplyr} environment.

# Assign tables to dplyr SQL connections using tbl()
tables <- DBI::dbListTables(conn)
for (table in tables) {
  assign(table, tbl(conn, table))
}

Squaring Up Population Data Across Governments

With the code in the chunk above, we set up a SQLite Connection object in our RStudio Global Environment pane to represent each table in the database like any other data.frame. The key difference is that the data in a connection object is a “lazy query” displayed in 1,000 row samples and is not brought into our local environment until we collect() it. Because of this feature, we can perform our joins and aggregations with other objects within the database, and only bring back the needed data into our environment. This enables us to explore without the memory constraints we have sometimes experienced even at this size of data, and also to experiment with generating complicated {SQL} queries.

head(select(local, 1:10), 5)

# Source:   lazy query [?? x 10]
# Database: sqlite 3.34.1
#   [/Users/davidlucey/Desktop/David/Projects/govFin/inst/extdata/gov_census.db]
        id year4 fy_end_date sch_lev_code function_code total_revenue
     <int> <int>       <int>        <int>         <int>         <int>
1 12001001  1972         930           NA            NA            17
2 12001001  1977         930           NA            NA            59
3 12001001  1979         930           NA            NA            66
4 12001001  1981         930           NA            NA            82
5 12001001  1982         930           NA            NA            92
# … with 4 more variables: total_rev_own_sources <int>, general_revenue <int>,
#   gen_rev_own_sources <int>, total_taxes <int>

In order to look at per capita statistics without double counting, we would like to check if the data duplicates population counts across tables in the various government entities. In the query below, we remove the Federal Government rows from our id_cols table, because those only go up until 1995, leaving all other government entities. In the output below, the aggregated population counts look about right at the state level (state code = 0), but it might also be interesting to check if the town-by-town aggregations add up to the state totals.

Click to see code generating output

state_level <- 
  id_cols %>%
  filter(type_code == 0) %>%
  left_join(popu, by = "id") %>%
  filter(year4 == 2016) %>%
  group_by(state_code, year4) %>%
  summarize(
    name, 
    tot_state = sum(population, na.rm = TRUE),
    .groups = "drop") %>% 
  select(name, state_code, tot_state)

# Source:   lazy query [?? x 3]
# Database: sqlite 3.34.1
#   [/Users/davidlucey/Desktop/David/Projects/govFin/inst/extdata/gov_census.db]
  name        state_code tot_state
  <chr>            <int>     <int>
1 ALABAMA              1   4858979
2 ALASKA               2    738432
3 ARIZONA              3   6828065
4 ARKANSAS             4   2978204
5 CALIFORNIA           5  39144818
6 COLORADO             6   5456574
7 CONNECTICUT          7   3590886

In the code below, we add up any populations tied to the County, Municipal, Township and other districts (type_code from 1-5). We live in Connecticut which doesn’t have county governments, but other states may have different structures, so we need to look across county, municipality, townships, school and special districts, so we are talking about a pretty complicated query including joins, filtering, pivots and aggregation. Now we can see that this is getting into some really complicated {SQL} for a guy who doesn’t know {SQL}. We haven’t added the state name yet, but the aggregations by state_code of the local totals are shown below.

Click to see code generating output

local_level <-
  id_cols %>%
  filter(type_code %in% c(1:5)) %>%
  select(id, state_code, state_code, type_code) %>% 
  left_join(popu) %>%
  filter(year4 == 2016) %>%
  select(state_code, type_code, population) %>% 
  pivot_wider(
    names_from = type_code,
    values_from = population,
    values_fill = 0,
    values_fn = sum
  ) %>% 
  rename(
    state_code = 1 ,
    county = 2,
    munis = 3,
    school = 4,
    special = 5,
    townships = 6) %>% 
  select(state_code, county, munis, townships, school, special) %>%
  mutate(tot_local = county + munis + townships + school + special)

# Source:   lazy query [?? x 7]
# Database: sqlite 3.34.1
#   [/Users/davidlucey/Desktop/David/Projects/govFin/inst/extdata/gov_census.db]
   state_code   county    munis townships school special tot_local
        <int>    <dbl>    <dbl>     <dbl>  <dbl>   <dbl>     <dbl>
 1          1  3633629  2067036         0      0       0   5700665
 2          2   319022   470256         0      0       0    789278
 3          3  6828065  5044676         0      0       0  11872741
 4          4  2468014  1444504         0      0       0   3912518
 5          5 38280002 15706524         0      0       0  53986526
 6          6  4560451  3170808         0      0       0   7731259
 7          7        0  1376121   2080762      0       0   3456883
 8          8   945934   268772         0      0       0   1214706
 9          9        0   672228         0      0       0    672228
10         10 19358262  6145441         0      0       0  25503703
# … with more rows

Adding the state name and joining the state_level and local_level tables, the summary table below compares the state populations with the aggregate local counts. The total population at the state level is generally in the range, but often considerably lower than “state” totals. It looks like we will have to aggregate up to the state level in order to calculate any per population amounts and not rely on the local totals.

Click to see code generating output

# Save query
pop_comp <-
  state_level %>%
  left_join(local_level, by = "state_code") %>%
  select(name, county, munis, townships, tot_local, tot_state) %>%
  mutate(percent_diff = tot_local/tot_state - 1)

# Collect query for display locally 
population_DT <- 
  pop_comp %>% 
  collect %>%
  mutate_at(vars(county:tot_state), function(x) x/1000)

Figure 1: Population Data from Local Tables vs State Aggregates (Thousands)

A shown in in the code chunk below, this is a complicated query for a relatively novice {SQL} and infrequent {dplyr} user, but was easily put together in a couple of hours. It was also easier to experiment and seemed less likely that we would make an error than if working in {SQL}. Though it may not be fully optimized, it seems like it could take a long time to learn SQL at that level. The {dbplyr} package has the sql_optimise() function, but it didn’t change our query at all. It will be interesting to see if tools develop to help users further optimize queries. As habitual {data.table} users, a similar translation layer would also be a big help.

# View SQL query
show_query(pop_comp)

<SQL>
SELECT `name`, `county`, `munis`, `townships`, `tot_local`, `tot_state`, `tot_local` / `tot_state` - 1.0 AS `percent_diff`
FROM (SELECT `name`, `LHS`.`state_code` AS `state_code`, `tot_state`, `county`, `munis`, `townships`, `school`, `special`, `tot_local`
FROM (SELECT `name`, `state_code`, `tot_state`
FROM (SELECT `state_code`, `year4`, `name`, SUM(`population`) AS `tot_state`
FROM (SELECT `LHS`.`id` AS `id`, `state_code`, `type_code`, `county`, `name`, `fips_code_state`, `fips_county`, `fips_place`, `fips_combined`, `year4`, `year_pop`, `population`, `enrollment`
FROM (SELECT *
FROM `id_cols`
WHERE (`type_code` = 0.0)) AS `LHS`
LEFT JOIN `popu` AS `RHS`
ON (`LHS`.`id` = `RHS`.`id`)
)
WHERE (`year4` = 2016.0)
GROUP BY `state_code`, `year4`)) AS `LHS`
LEFT JOIN (SELECT `state_code`, `county`, `munis`, `townships`, `school`, `special`, `county` + `munis` + `townships` + `school` + `special` AS `tot_local`
FROM (SELECT `state_code`, `1` AS `county`, `2` AS `munis`, `3` AS `townships`, `4` AS `school`, `5` AS `special`
FROM (SELECT `state_code`, SUM(CASE WHEN (`type_code` = 1) THEN (`population`) WHEN NOT(`type_code` = 1) THEN (0.0) END) AS `1`, SUM(CASE WHEN (`type_code` = 2) THEN (`population`) WHEN NOT(`type_code` = 2) THEN (0.0) END) AS `2`, SUM(CASE WHEN (`type_code` = 4) THEN (`population`) WHEN NOT(`type_code` = 4) THEN (0.0) END) AS `4`, SUM(CASE WHEN (`type_code` = 5) THEN (`population`) WHEN NOT(`type_code` = 5) THEN (0.0) END) AS `5`, SUM(CASE WHEN (`type_code` = 3) THEN (`population`) WHEN NOT(`type_code` = 3) THEN (0.0) END) AS `3`
FROM (SELECT `state_code`, `type_code`, `population`
FROM (SELECT `LHS`.`id` AS `id`, `state_code`, `type_code`, `year4`, `year_pop`, `population`, `enrollment`
FROM (SELECT `id`, `state_code`, `type_code`
FROM `id_cols`
WHERE (`type_code` IN (1, 2, 3, 4, 5))) AS `LHS`
LEFT JOIN `popu` AS `RHS`
ON (`LHS`.`id` = `RHS`.`id`)
)
WHERE (`year4` = 2016.0))
GROUP BY `state_code`))) AS `RHS`
ON (`LHS`.`state_code` = `RHS`.`state_code`)
)

Aggregating All Income and Spending by Population

To take it to another level, now we can get back to the original challenge: calculating the state total income and spending per population by all forms of government. In order to do this, we have to merge the 5 non-federal government tables together, aggregate the selected items, join on the identifier and population tables and filter for only full census years (ending in 2 or 7). We also then calculate the per population amounts for and tidy up our three selected variables, which just about doubles the query size.

Click to see code generating output

# State population and identifier SQL Connection object
state_pop <- popu %>%
  left_join(id_cols) %>%
  filter(type_code == 0) %>%
  select(state_code, year4, population, name)

Joining, by = "id"

# Fields to keep from tables
fields <-
  c("id",
    "total_revenue",
    "total_taxes",
    "total_expenditure",
    "year4")

# Years to keep
full_census <- seq(1972, 2017, 5)

# Union of tables 
all_govs <- 
  union_all(
    select(county, fields), 
    select(local, fields), 
    select(state, fields), 
    select(special, fields),
    select(school, fields)
    )

Note: Using an external vector in selections is ambiguous.
ℹ Use `all_of(fields)` instead of `fields` to silence this message.
ℹ See <https://tidyselect.r-lib.org/reference/faq-external-vector.html>.
This message is displayed once per session.

# Aggregations after merging tables
vars <- c("year4", "name", "rev_pop", "tax_pop", "spend_pop")

# Final SQL connection object
state_agg <- all_govs %>%
  left_join(select(id_cols, c("id", "state_code"))) %>%
  group_by(state_code, year4) %>%
  summarize_at(vars(total_revenue:total_expenditure), sum, na.rm = TRUE) %>%
  ungroup() %>%
  left_join(state_pop, by = c("state_code", "year4")) %>%
  mutate(
    rev_pop = total_revenue * 1000 / population,
    tax_pop = total_taxes * 1000 / population,
    spend_pop = total_expenditure * 1000 / population
  ) %>%
  select(any_of(vars)) %>%
  tidyr::pivot_longer(names_to = "variable",
                      cols = contains("pop")) %>%
  filter(year4 %in% full_census)

Joining, by = "id"

# View SQL query
dplyr::show_query(state_agg)

<SQL>
SELECT *
FROM (SELECT `year4`, `name`, 'rev_pop' AS `variable`, `rev_pop` AS `value`
FROM (SELECT `year4`, `name`, `total_revenue` * 1000.0 / `population` AS `rev_pop`, `total_taxes` * 1000.0 / `population` AS `tax_pop`, `total_expenditure` * 1000.0 / `population` AS `spend_pop`
FROM (SELECT `LHS`.`state_code` AS `state_code`, `LHS`.`year4` AS `year4`, `total_revenue`, `total_taxes`, `total_expenditure`, `population`, `name`
FROM (SELECT `state_code`, `year4`, SUM(`total_revenue`) AS `total_revenue`, SUM(`total_taxes`) AS `total_taxes`, SUM(`total_expenditure`) AS `total_expenditure`
FROM (SELECT `LHS`.`id` AS `id`, `total_revenue`, `total_taxes`, `total_expenditure`, `year4`, `state_code`
FROM (SELECT `id`, `total_revenue`, `total_taxes`, `total_expenditure`, `year4`
FROM `county`
UNION ALL
SELECT `id`, `total_revenue`, `total_taxes`, `total_expenditure`, `year4`
FROM `local`) AS `LHS`
LEFT JOIN (SELECT `id`, `state_code`
FROM `id_cols`) AS `RHS`
ON (`LHS`.`id` = `RHS`.`id`)
)
GROUP BY `state_code`, `year4`) AS `LHS`
LEFT JOIN (SELECT `state_code`, `year4`, `population`, `name`
FROM (SELECT `LHS`.`id` AS `id`, `year4`, `year_pop`, `population`, `enrollment`, `state_code`, `type_code`, `county`, `name`, `fips_code_state`, `fips_county`, `fips_place`, `fips_combined`
FROM `popu` AS `LHS`
LEFT JOIN `id_cols` AS `RHS`
ON (`LHS`.`id` = `RHS`.`id`)
)
WHERE (`type_code` = 0.0)) AS `RHS`
ON (`LHS`.`state_code` = `RHS`.`state_code` AND `LHS`.`year4` = `RHS`.`year4`)
))
UNION ALL
SELECT `year4`, `name`, 'tax_pop' AS `variable`, `tax_pop` AS `value`
FROM (SELECT `year4`, `name`, `total_revenue` * 1000.0 / `population` AS `rev_pop`, `total_taxes` * 1000.0 / `population` AS `tax_pop`, `total_expenditure` * 1000.0 / `population` AS `spend_pop`
FROM (SELECT `LHS`.`state_code` AS `state_code`, `LHS`.`year4` AS `year4`, `total_revenue`, `total_taxes`, `total_expenditure`, `population`, `name`
FROM (SELECT `state_code`, `year4`, SUM(`total_revenue`) AS `total_revenue`, SUM(`total_taxes`) AS `total_taxes`, SUM(`total_expenditure`) AS `total_expenditure`
FROM (SELECT `LHS`.`id` AS `id`, `total_revenue`, `total_taxes`, `total_expenditure`, `year4`, `state_code`
FROM (SELECT `id`, `total_revenue`, `total_taxes`, `total_expenditure`, `year4`
FROM `county`
UNION ALL
SELECT `id`, `total_revenue`, `total_taxes`, `total_expenditure`, `year4`
FROM `local`) AS `LHS`
LEFT JOIN (SELECT `id`, `state_code`
FROM `id_cols`) AS `RHS`
ON (`LHS`.`id` = `RHS`.`id`)
)
GROUP BY `state_code`, `year4`) AS `LHS`
LEFT JOIN (SELECT `state_code`, `year4`, `population`, `name`
FROM (SELECT `LHS`.`id` AS `id`, `year4`, `year_pop`, `population`, `enrollment`, `state_code`, `type_code`, `county`, `name`, `fips_code_state`, `fips_county`, `fips_place`, `fips_combined`
FROM `popu` AS `LHS`
LEFT JOIN `id_cols` AS `RHS`
ON (`LHS`.`id` = `RHS`.`id`)
)
WHERE (`type_code` = 0.0)) AS `RHS`
ON (`LHS`.`state_code` = `RHS`.`state_code` AND `LHS`.`year4` = `RHS`.`year4`)
))
UNION ALL
SELECT `year4`, `name`, 'spend_pop' AS `variable`, `spend_pop` AS `value`
FROM (SELECT `year4`, `name`, `total_revenue` * 1000.0 / `population` AS `rev_pop`, `total_taxes` * 1000.0 / `population` AS `tax_pop`, `total_expenditure` * 1000.0 / `population` AS `spend_pop`
FROM (SELECT `LHS`.`state_code` AS `state_code`, `LHS`.`year4` AS `year4`, `total_revenue`, `total_taxes`, `total_expenditure`, `population`, `name`
FROM (SELECT `state_code`, `year4`, SUM(`total_revenue`) AS `total_revenue`, SUM(`total_taxes`) AS `total_taxes`, SUM(`total_expenditure`) AS `total_expenditure`
FROM (SELECT `LHS`.`id` AS `id`, `total_revenue`, `total_taxes`, `total_expenditure`, `year4`, `state_code`
FROM (SELECT `id`, `total_revenue`, `total_taxes`, `total_expenditure`, `year4`
FROM `county`
UNION ALL
SELECT `id`, `total_revenue`, `total_taxes`, `total_expenditure`, `year4`
FROM `local`) AS `LHS`
LEFT JOIN (SELECT `id`, `state_code`
FROM `id_cols`) AS `RHS`
ON (`LHS`.`id` = `RHS`.`id`)
)
GROUP BY `state_code`, `year4`) AS `LHS`
LEFT JOIN (SELECT `state_code`, `year4`, `population`, `name`
FROM (SELECT `LHS`.`id` AS `id`, `year4`, `year_pop`, `population`, `enrollment`, `state_code`, `type_code`, `county`, `name`, `fips_code_state`, `fips_county`, `fips_place`, `fips_combined`
FROM `popu` AS `LHS`
LEFT JOIN `id_cols` AS `RHS`
ON (`LHS`.`id` = `RHS`.`id`)
)
WHERE (`type_code` = 0.0)) AS `RHS`
ON (`LHS`.`state_code` = `RHS`.`state_code` AND `LHS`.`year4` = `RHS`.`year4`)
)))
WHERE (`year4` IN (1972.0, 1977.0, 1982.0, 1987.0, 1992.0, 1997.0, 2002.0, 2007.0, 2012.0, 2017.0))

When we run this query against the database, the response is pretty close to instantaneous, and we still haven’t even taken the data into memory when we plot it below with ggplot.

Click to see code generating output

# Prepare plot directly from our SQL Connection object
p <- ggplotly(
  state_agg %>%
    ggplot(aes(year4, 
               value, 
               group = name, 
               color = name)
           )+
    geom_line() +
    facet_wrap( ~variable) +
    scale_y_continuous(labels = scales::dollar,
                       trans = "log10") +
    theme_bw() +
    theme(legend.position = "none") +
    labs(
      title = "State Select Revenue and Spending Items - 1972-2017",
      x = "Year",
      y = "Amount per Population"
    )
)

As a final step, we can add our two queries as “Views” in the database so that they are immediately available in the future.

Click to see code generating output

# SQL render 
q <- dbplyr::sql_render(pop_comp)
q1 <- dbplyr::sql_render(state_agg)

# Add SQL queries as views in database
view_pop_comp <- paste("CREATE VIEW pop_comp AS", q)
view_state_agg <- paste("CREATE VIEW state_agg AS", q1)
rs <- DBI::dbSendQuery(conn, view_pop_comp)
dbClearResult(rs)
rs <- DBI::dbSendQuery(conn, view_state_agg)
dbClearResult(rs)

# Showing views now in database
DBI::dbListTables(conn)

 [1] "county"    "federal"   "id_cols"   "local"     "pop_comp"  "popu"     
 [7] "school"    "special"   "state"     "state_agg"

Conclusion

The need to know {SQL} in the data profession is widely agreed upon, but there is so much to learn. This post might show that spending some of that time getting really good at {dplyr} might offer 2 skills for the price of 1. Complicated joins, aggregations and indexing may not be beginner data manipulations, but given how easy this was at this stage, it might have been better if we had been nudged away from .csv’s earlier. Now at a more advanced levels, the missing pieces we would like to see are mainly a translation layer to use {data.table} to interact with the connection object and also that the ability to use regular expressions with {stringr}, which don’t seem to be implemented yet.

In Search of a Better Home Price Prediction in Greenwich, CT - Part 1

Thu, 10 Dec 2020 00:00:00 +0000

Figure 1: Average Single Family Homes in Towns of Connecticut 1999-2018

Introduction

Though losing ground in recent years, Connecticut has long had some of the highest average incomes and home prices in the country. Within the State, some towns have had significantly higher selling prices than others (shown in chart above). In this series, we will attempt to build a model to predict selling prices in the highest-priced Town of Greenwich (shown in red). In the case of Greenwich, the same house around the corner in a neighboring town, might sell for a considerably lower price because of perceived advantages, such as schools quality, tax rates and other local amenities. Even within the town, a small group of neighborhoods have average selling prices which are significantly higher than others.

To make it even more difficult, some preferences have been shifting in recent years as newcomers have favored smaller plots in closer proximity to shops and transportation. In addition, along with the income of town residents in aggregate and Connecticut’s financial challenges more broadly, property prices have been slowly declining since 2007. In 2017, after some commentators disparaged local selling conditions in the national press, liquidity dried up and the number of units sold fell to levels not seen since the financial crisis. The market only started picking up again in 2019, and accelerated further in 2020, as Millennial families, who had deferred normal patterns of suburban migration, finally took a serious look in hopes of riding out the NYC lock-down with a back yard.

In this series (of yet undetermined size), we will explore in more depth the prices of homes in Greenwich over time and in the recent period. From time to time, we have noticed that websites offering algorithms on millions of homes nationally have surprisingly large errors when “predicting” local selling prices. By its nature, real estate is an emotional purchase and market conditions fluctuate and complete and clean data is hard to come by, so there is a limit to the accuracy of such predictions, but we will explore if it is possible to do better in this series.

Resources and Preparation

As described in Using drake for ETL and building Shiny app for 900k CT real estate sales (also the source of the graphic above), we did all of the ETL and modeling for this series using the drake package. Although it was bulk of the work, we are not going to discuss data preparation in this series. We tried on and off to wrangle transaction data from multiple sources for some time. Only with the addition of drake, we felt we were able to get clean enough data for modeling (though still not perfect). Follow the link to the full Shiny app of Connecticut Property Selling Prices vs Assessment Values over Three Revaluation Cycles shown above for how a {drake} workflow looks.

As this will be the first attempt by Redwall Analytics blog site to build a model on data, there is much to learn. As usual, we looked around for other posts and books attempting to predict real estate prices, and there were not a lot. Hands-On Machine Learning with R by Bradley Boehmke & Brandon Greenwell was an excellent resource on the options for modeling home prices with regression, but this referred to Kaggle’s Ames data set, which doesn’t have the time span or heterogeneity of our data.

In this post, we heavily use the {DataExplorer} package. In Part 3, we will also rely on the {vtreat} package for data complexities like handling missing-ness, categorical variables with too many or rare levels and for variable selection. In Part 4, we will use the {h2o} auto_ML() functionality to explore and look for the best model.

Data Summary

We have gathered 17 years of Single Family home sales, and tried to filter out transactions which might not be “arms length”. For example, we edited out transactions which took more than one year to close and sold for less than 75% of original listing price assuming the normal selling process had been impaired. By convention in Connecticut, real estate is assessed at 70% of estimated market value. We filtered out homes which sold for less than 75% or greater than 180% of assessed value (ie: 52.5-126% of the Town’s best guess for tax purposes), making the assumption that there must have been special circumstances to warrant such a wide divergence.

In the recent slow period, it is likely that many homes came to market and were not sold on the first attempt. We are also only able to consider homes which actually sold, and are not adjusting days on the market where more than one attempt was made before selling. We also note that some of our data fields, such as garage, taxcard, court, pool and condo, age_renov and age_add, are only available in the last few years. Others like rooms is not available more recently though we have beds and bath consistently over the period.

Out of approximately 16,000 total property sales and 12,000 Single Family home sales during the period, we have just over 7,100 for our model building. In its 5-year revaluation conducted in 2015, we observed that the Town of Greenwich used about half of all available property sales, so it is likely that there are still transactions which can be considered non-representative, such as related parties, financial distress, and other special conditions, like homes which were purchased for demolition. After all that, it is likely that there are outlier transactions in our data set which will need further consideration if the goal is to predict the selling price of a single family home under normal circumstances.

Historical and Recent Trends Since 2003

In recent years, the pace of sales slowed sharply, and average prices and sizes of homes sold, declined from 2007 peak levels. There has been much discussion of rising income inequality, but as we wrote in IRS Data Shows Growth in Number not Income of Highest Earners since 2005, income differentials between the highest and lowest earning CT groups have narrowed somewhat as conditions changed, particularly in the financial services industry, where many firms departed or closed since the GFC, and new equally high paying firms did not replace them.

The values of the most expensive properties have been in slow decline, while the least expensive have risen consistently since 2012, and may now exceed the previous peak levels. Even before CV-19, even with steady growth at the lower end, the average price per square foot of single family homes sold fell from over $675 to around $550. Within the overall downtrend, there have also been a change in the age and size mix, as builders replaced usually older and smaller, with larger homes on existing lots. Despite abundant new building, the average age of homes sold in the sample has risen by 5-7 years over the mid-2000s. Contributing to this, tastes have shifted from multi-acre plots in the “Back Country” to smaller lots closer to town and transportation, often in the “Coastal” area.

It is likely that the many owners of larger, more expensive properties far from the center, decided to hold onto them rather than sell in disadvantageous conditions. The decline in volumes shown in 2017 and 2018 and average square foot of homes sold may be partly a function of this. It most housing markets, people tend to sell in order to move to a new home, but this need can be less urgent for some in Greenwich. While the market was already picking up in 2019 and 2020, average prices may have been held back by backlog of demand to sell, which is likely to run out eventually. In short, the median and average prices probably didn’t represent the true run rate of mean valuation a couple of years ago, nor does it in 2020. The coming year will be interesting to watch.

Click to see code generating table

#summary stats in datatable
t <- 
 unique(data)[
   year(date_sold) > 2000, 
   .(
     .N,
     mn_sqft = mean(sqft, na.rm = TRUE),
     mn_price = mean(price, na.rm = TRUE),
     med_price = median(price, na.rm = TRUE),
     mn_price_sqft = (sum(price, na.rm = TRUE) / sum(sqft, na.rm = TRUE)) * 1000,
     mn_age = mean(age, na.rm = TRUE)
   ), year(date_sold)][
   ][order(year)]

# Print
dt <- 
  DT::datatable(
    t,
    colnames =
      c(
        "Year",
        "Annual Units",
        "Avg. Sqft.",
        "Avg. Price ($k)",
        "Median Price ($k)",
        "Avg Price/Sqft. ($)",
        "Avg. Age"
      ),
    rownames = FALSE,
    options = 
      list(
        pageLength = 18,
        scrollY = TRUE)
  ) %>%
  formatRound(columns = 1,
              digits = 0) %>%
  formatRound(columns = c(2:7),
              mark = ",",
              digits = 0)

Figure 2: Summary of Subset of Greenwich, CT Home Sale Statistics from 03-2020

Assessed Values

Key variables for our analysis are not hard to find, with assessments, available ex ante for model building, looking like a front-runner. The “Overall Quality” variable was one of the most important in the Kaggle Ames data, but we don’t have an equivalent measure for recent periods. Every five years, the town visits homes in order to determine the “condition” and “grade” when setting the portion of value attributable to “improvements” for tax purposes. This portion of the assessed value would ideally be our proxy for “Overall Quality”, but we only have it for the period of 2010-18. Since there are ~14,000 single family homes, and we can only remember someone from the Assessor’s office showing up once in 20 years, so this may also not be the most consistent and accurate measurement. We do however have the two components of the assessed value, land and improvements, up until the most recent sales. In Greenwich, the ratio of improvements to total assessed value is only 50% on average. In comparison, we discovered that improvements made up 85% of assessed value in one area in Florida so the size and attributes of the physical structure is likely to be less than in most places. For our analysis, we will subdivide assessed value into its land (assess_land) and improvement (assess_improve) components in hopes that the portion attributable to improvements can be a proxy for “Overall Quality” though imperfect.

Correlations over Early Years

Over the 2003-2016 period shown in Figure 3, we can see that sqft is likely to be the most important variable for predicting price, with 88% positive correlation (shown in red), but related variables like beds and bath are also highly correlated. Similarly, assess_land and assess_improve are positively correlated, though less than sqft. We are focusing on prediction in this series, but several of our many important independent variables are highly correlated with each other, which would cause problems if our focus was inference. The Floor Area Ratio (far) is a function of acre and zoning (not shown here). FAR has become an increasingly important consideration over the period, as it governs the maximum size of home which can be built on a given lot. With the move towards smaller lots in the Coastal areas, builders scooped up lots perceived to be less than the best use of that land. Not surprisingly, age is negatively correlated (older homes are less desired). Older homes sold also have fewer bathrooms, bedrooms, square feet and even acreage shown in blue. The higher correlation of price with bath than beds is interesting, but this may be a function of age. Newer homes probably are built with more bathrooms on balance.

Days on market (dom) is only slightly correlated with price, but we have filtered out the really stale listings (over 365 days), so it would probably look different if we had all sales. We can also see that dom is negatively correlated to sp_olp (selling price divided by original listing price). Homes which are on the market longer sell for relatively less compared to the original listing price as prices are reduced. The Assessment Ratio (AR), the ratio of selling price to assessed value, should be unbiased with respect to selling prices, but we can see that the correlation is somewhat positive during this period. This suggests that homes with higher priced homes were sold for relatively more versus assessed value than would have been expected (ie: they were under-assessed). We can also see that AR is positively correlated to sqft, beds, bath, and negatively correlated to age. It suggests that the assessment process may have not fully accounted on the tax rolls for newer, larger homes, at least in the population which was sold, at least up until 2016.

Click to see code generating table

# Select vars
late_vars <- 
  c("condo", "court", "pool", "garage", "taxcard", "qtr_median")
chart_vars <- 
  setdiff(vars, late_vars)

# Separate continuous variables and log transform where appropriate
cont <- 
  split_columns(copy(data[between(year(date_sold), 2004, 2016), ..chart_vars]))$continuous

# Helper to filter skewed vars
skewed <- function(var) abs(e1071::skewness(var, na.rm = TRUE)) > 1
is_skewed <- names(cont)[sapply(cont, skewed)]
is_skewed <- setdiff(is_skewed, "bath")
cont[, (is_skewed) := lapply(.SD, log1p), .SDcols = is_skewed]

# Corrplot of 2004-2016
p <- plot_correlation(
  cont, 
  type = "continuous", 
  cor_args = list("use" = "pairwise.complete.obs"),
  title = "Correlation Plot of Key Variables - 2004-2016",
  ggtheme = theme_bw()
  )

Figure 3: Pairwise Correlations of Key Numeric Variables from 2003-2016

Correlations Since 2017

In Figure 4 below, we isolate the years 2017-20 where we have additional variables including: age_renov (years since renovation), age_add (years since addition), court, pool, garage (number of garages) and taxcard (source of sqft data). There don’t seem to be many significant differences in the correlations of key variables linked to price from the earlier period in Figure 3 above. One key difference though, is that the Assessor’s Office seems to have adjusted assessed values to better reflect selling prices in the 2015 revaluations. Selling prices of larger homes are now negatively correlated with Assessed Value. Not surprisingly, age_renov (age of renovation) and age_add (age of addition), which we didn’t have for the early period, are the similarly negatively correlated with price as above. age now is essentially uncorrelated to AR. We will see in Figure ?? below, the price relationship with age may be “multimodal”. If we were to just look at taxcard (the source of sqft), it would have a positive correlation of 20% with price and sqft. Builders and architects reporting newer homes are likely larger and more expensive.

Click to see code generating table

# Select vars
chart_vars <- 
  setdiff(vars, c("qtr_median", "condo"))

# Separate continuous variables and log transform where appropriate
cont <- 
  split_columns(copy(data[year(date_sold) > 2016, ..chart_vars]))$continuous
cont[, (is_skewed) := lapply(.SD, log1p), .SDcols = is_skewed]

# Corrplot of 2017-2010
p <- plot_correlation(
  cont, 
  type = "continuous", 
  cor_args = list("use" = "pairwise.complete.obs"),
  title = "Correlation Plot of Key Variables - 2017-2020",
  ggtheme = theme_bw()
  )

Figure 4: Pairwise Correlations of Key Numeric Variables from 2017-2020

Histograms

There is a lot to be discovered from the histograms of our property sales. In the charts below, we removed home sold in 2004 because that was the last year before selling prices stepped up to higher levels. Our star variables, assess_land, far and sqft, all look Gaussian on log scale, although some might be moderately skewed. We are putting a lot of years together here, so that might be part of it. It is interesting that assess_land and assess_improve are more skewed than the other variables, and also that assess_land is positively, while assess_building is negatively skewed. There is possibly a lower boundary for assess_land, and also waterfront properties may fatten the tails at the higher end. Also, there is a lot more variation in assess_improve than in assess_land even though both make up half of the total assessed value on balance. All of the lots in a given zoning and acreage should have similar values so that makes sense. The majority of home sales are less than 1 acre, though there are a few areas zoned for up to 10 acres. The acre variable looks less bell shaped, so we will have to give this some consideration. The price variable looks right skewed, but only moderately so.

Click to see code generating charts

# Select vars
chart_vars <- setdiff(is_skewed, c("dom", "age_renov", "num_sales"))

# Separate continuous variables and log transform where appropriate
cont <- 
  split_columns(copy(data[between(year(date_sold), 2004, 2020), ..chart_vars]))$continuous

# Plot histogram of skewed distributions log10 transformed
p <- plot_histogram(
  cont[, .SD, .SDcols = chart_vars],
  scale_x = "log10",
  title = "Histograms of Key Numeric Variables - Log Scale",
  ggtheme = theme_bw())

Figure 5: Histograms of Key Numeric Variables on Log10 Scale from 2003-2020

As mentioned above, most are “nearly normal”, but acre, assess_improve and assess_land show moderate skew. One place we are still not clear is how much we should worry about the remaining skew affecting our predictions. It seems likely that it would be possible to further reduce with Box Cox transformations, or dropping outliers. As we will discuss further down, further transformation of acre may be needed.

Click to see code generating charts

# Select vars
trans_vars <- setdiff(is_skewed, c("bath", "frontage_ft", "num_sales"))

# Calc skew and kurtosis vectors
skew <- 
  apply(
    data[between(year(date_sold), 2004, 2020), sapply(.SD, log1p), .SDcols = trans_vars], 2, e1071::skewness, na.rm = TRUE
    )
kurt <-
  apply(
    data[between(year(date_sold), 2004, 2020), sapply(.SD, log1p), .SDcols = trans_vars], 2, e1071::kurtosis, na.rm = TRUE
    )

# Display skew and kurtosis dt
d <- 
  data.table(
    measure = c("skew", "kurtosis"),
    rbind(skew[complete.cases(skew)],
          kurt[complete.cases(kurt)])
  )

Most homes were built between 50-100 years ago, though we can see the recent (re)building boom close to the y-axis. Renovations and additions mostly over the 20 and 40 years, respectively. It is hard to see here, but most homes sold have not had recent additions or renovations. None of the age variables appear to be bell-shaped, so this will take some further thinking. We filtered out sales with Assessed Ratios (AR) above 1.8 and below 0.75, homes which took longer than 365 days (dom) to sell and homes which sold for less than 75% of original list, so these charts are all truncated, and even if they were not, have natural upper and lower boundaries. The variables for beds, bath and garage all look bell-shaped, but we are expecting to make some transformations of these because they are so highly correlated with other variables.

Click to see code generating charts

# Separate continuous variables and log transform where appropriate
cont <- 
  split_columns(copy(data[between(year(date_sold), 2004, 2020)]))$continuous

# Choose chart vars
chart_vars <- names(cont[, .SD, .SDcols = !is_skewed])

# Plot numeric not log transformed data
p <- plot_histogram(
  cont[, .SD, .SDcols = chart_vars],
  title = "Histograms of Key Numeric Variables - Ordinal Scale",
  ncol = 3L, 
  nrow = 4L,
  ggtheme = theme_bw())

Figure 6: Histograms of Key Numeric Variables on Ordinal Scale from 2004-20

Scatter Plots

In the chart below, we show the log-log scatter plot of these our key variables versus price (y-axis), which shows selling prices flatten out after ~1 acre and possibly with assess_land, though less defined. Variables pertaining to house size like sqft and far continue their linear rise over the full range of prices. If anything, assess_improve looks like sqft, but might even have a slight quadratic shape. We might experiment with a splines to represent acre to see how that goes.

Click to see code generating charts

# Select vars
scatter_vars <- setdiff(is_skewed, c("court", "taxcard", "num_sales"))

# Scatterplot
p <- plot_scatterplot(
  data[, .SD, .SDcols = scatter_vars], 
  by = "price",
  scale_y = "log10",
  scale_x = "log10",
  ncol = 3L, 
  nrow = 2L,
  sampled_rows = 1000L,
  theme_config = list("axis.text.x" = element_text(angle = 90)),
  title = "Scatter Plot of Sample of Logged Numeric Variables by Log Price 2004-20",
  ggtheme = theme_bw()
  )

Figure 7: Scatter of Sample of Key Numeric Variables from 2004-2020

The next group of scatter plots shown below are time-related variables. The newest homes sell for the highest prices, but there appears to be some homes that hold up beyond 50 years, then several higher points before falling off again after 110 (although the data gets thin at this point). There are a considerably number of 200+ year old homes sold, but these may be historical landmarks. At least recorded in the data, there is a surge of renovations in the last 20 years, which seems to add considerably to price, though less relevant to price after about 25 years. Additions don’t seem add as noticeably to price as renovations.

The dates are data.table integer types, so the year is not visible, but there is a very wide range of selling prices represented consistently over time. A slight decline in number of higher priced sales in recent years is apparent, and there are two periods with fewer very high priced sales around the GFC and again around 2017. Homes selling below $1 million are minority over time and there have been fewer in the recent years. The dom variable has been truncated after 360 days, but would fall off if the chart continued.

Click to see code generating charts

# Select vars
chart_vars <- c("price", "age", "age_add", "age_renov", "date_sold", "dom")

# scaterplot
p <- plot_scatterplot(
  data[, ..chart_vars], 
  by = "price",
  ncol=3L, 
  scale_x = "log10",
  theme_config = list("axis.text.x" = element_text(angle = 90)),
  title = "Scatterplot of Time-Related Variables by Log Price 2004-20",
  ggtheme = theme_bw()
  )

Figure 8: Scatter Plot of Sample of Time-Related Variables from 2004-20

The convention in Connecticut is that assessed value should represent 70% of the market value, so on balance, homes should sell for about 1.4 assessed value (1 / 0.7), and we have filtered out AR below 0.75 and above 1.8. The chart seems to peak around this level, but it is hard to say much about density in this chart which is sampled. We also see that some homes have turned over every few years, and these seem to have lower prices though above “num_sales” was slightly positively correlated with price. Also, we can see that the highest valued home sales sell for less than the original listing price. On balance, homes which are selling above original list are towards the lower end of sales prices, though we filtered below 75% of original list price when they took more than a year to sell.

Click to see code generating charts

# Select vars
chart_vars <- c("price", "sp_olp", "AR", "num_sales")

# Scatterplot
p <- plot_scatterplot(
  data[, ..chart_vars], 
  by = "price",
  #sampled_rows = 2000L,
  theme_config = list("axis.text.x" = element_text(angle = 90)),
  title = "Scatterplot of Unlogged Numeric Variables by Log Price 2004-20",
  ggtheme = theme_bw()
  )

Figure 9: Scatter Plot of Sample of Other Variables Grouped by Price from 2003-2020

Catagorical Variables

There are over 50 neighborhoods defined by the Town, but we have divided these into 4 larger groups with “North of Coastal” being the largest by volume, but significantly lower selling prices on average. We will talk more about school and neighborhood in greater detail in the next post as cross-sections of location seems like an important aspect of this project. There are a few other geographical variables in our data, including elem, area and assess. For now, the four super-neighborhoods are defined in the legend, and the categorical variables are divided by location and aggregated by sale price in Figure 10 below.

Homes with <4 beds or <2.6 bathrooms were small, <5 beds or <6 baths were regular and above those numbers were large (denoted bed_factor and bath_factor). Home sales closing between May and August, when families try to prepare for the start of school, which were more than half the sales by volume, are marked as peak. Elementary schools in the four highest priced districts were denoted by school. The build variable refers to properties which have the potential to add 25% more space given their far, and we set up zng_levels based on lowest, middle and highest average sales prices. We thought that properties which sold often might sell for less, so we denoted properties selling 3 or more times as high_sales. Almost 1/4 of the properties by price sold this frequently in 17 years, which is surprisingly high.

In these charts, we can see that “North of Coastal” is less represented in the highly desired elementary school districts, and has a bigger share of smaller and older homes and less expensive zoning. Among homes which were sold, it has more on lots which could still be expanded or rebuilt into larger square footage.

Click to see code generating charts

# Separate discrete variables
disc <- split_columns(copy(data))$discrete

# Helper function to filter vars with less than 15 levels
low_levels <- function(var, levels = 15) length(levels(as.factor(var))) < levels

# Filter cols with find_levels and drop "street_suffix"
bar_data <-
  cbind(
    disc[!is.na(build) & bed_factor != "" & age_range != "", 
             .SD[, !"street_suffix_2"], .SDcols = low_levels], 
    price = log10(data$price)
    )

# Reorder
setcolorder(bar_data, names(bar_data)[c(1:5, 13, 6:12, 14)])

# Barplot
p <- plot_bar(
  bar_data,
  with = "price",
  by = "location",
  title = "Bar Plots of Key Categorical Variables by Price",
  nrow = 4L,
  ncol = 2L,
  ggtheme = theme_bw())

Figure 10: Bar Plot of Key Categorical Variables by Price and Location from 2003-2020

Figure 11: Bar Plot of Key Categorical Variables by Price from 2003-2020 (continued)

Conclusion

DataExplorer offered structure which forced us to think carefully about our variables from a number of perspectives. We also looked at its PCA and QQ-plots, but didn’t include them in this post, and may still use plot_boxplot() when we consider outliers in the upcoming posts. We learned that attributes of the actual structure of home should make up relatively less of the value than most places. Once log transformed, sqft, assess_improve and far are all linear and highly correlated with price. The age and size of homes interacts within location, so that an older home in a desired might actually be dragged downwards towards the value of the land it sits on, while this might be less pronounced in other locations. Newer homes in highly desired areas might be rewarded with even higher selling prices given the scarcity. We think there are interactions as we mentioned with age and bath, bed and taxcard so it may not be easy to specify a linear model which would yield the best predictions. Some variables, like acre and assess_land which may not be linear over price might benefit from transformations like a spline. We have some variables which run the full time span, and others which only are available starting in 2017. Since the overall prices have been somewhat stable over time, we also might expand or reduce the time periods which we use in order to have enough data in a given location. We have done some research in Bayesian Hierarchical models, which would seem to allow us to form a “prior” about the cross-sectional behavior of neighborhoods, so this might be enable us to use our 50 neighborhoods and still have enough data in periods to get accurate predictions. In the next post, we will look more closely about the affect of location on prices over time.

Tapping Yelp data with Apache Drill from Mac using {sergeant}

Tue, 27 Oct 2020 00:00:00 +0000

Click to see package details

# Libraries
packages <- 
  c("tidyverse",
    "sergeant",
    "tictoc"
    )

if (length(setdiff(packages,rownames(installed.packages()))) > 0) {
  install.packages(setdiff(packages, rownames(installed.packages())))  
}

invisible(lapply(packages, library, character.only = TRUE))

knitr::opts_chunk$set(
  comment = NA,
  fig.width = 12,
  fig.height = 8,
  out.width = '100%'
)

Introduction

At Redwall, we have been in nonstop exploration of new data sets over the last couple of years. As our data grows and the targets of interest get bigger, we have been finding the old method of loading csv’s from disc, and working on the full set in memory is becoming less optimal. We thought we would try Apache Drill via the {sergeant} package created by Bob Rudis, a prolific R developer. Apache Drill seems amazing because it would allow us to be agnostic as to data source and type. Usually, we write blog posts to show off things we have learned which are actually working. The punchline in this case though, is that we were not able to get where we hoped so far with Drill. We will chronicle what we have done so far, and where we are still falling short.

Recall in Finding the Dimensions of secdatabase.com from 2010-2020 - Part 2, we were able to query a data set which was over 20GB with the AWS Athena query service with pretty much instant response. With Apache Drill on one node on our 2005 Apple iMac with 8GB of RAM, queries with a couple of joins and some aggregation were taking at least 30 minutes, but usually much longer on a much smaller data set (if they didn’t crash our computer altogether). This could well be our machine, something we did wrong in configuring, poor query management or all of the above. We are writing this post in hopes of a response from experts, as well as to help others who might be trying to understand how to use Java, Drill or even the command line from RStudio. We promise to update the post with any feedback, so that it provides a pathway to others seeking to do the same.

Yelp Academic Data Set

We were hoping to avoid downloading it, but then we found Bob Rudis’ excellent Analyzing the Yelp Academic Dataset w/Drill & sergeant blog post and became intrigued by the possibility of having the flexible connection it offered, agnostic about storage and data formats. The Yelp Academic data set is about 10GB in size and took us over an hour to download, and are summarized in the image from the web page above. We hoped that we might be able to use it to explore the death rate of businesses in areas with differing COVID-19 mask and other non pharmaceutical interventions. Unfortunately, this is not possible at the moment, because it only runs through the end of 2019. The files are all in JSON format, and were one of the original examples given on the Apache Drill website and with the {sergeant} package. Shown below, the “business” file is the smallest, and “reviews” are by far the largest. Users visit businesses and give reviews, check-ins or tips, so the two main identifiers which tie the tables together are business_id and the user_id. There is a lot of opportunity to practice joins and aggregations if you can get it to work.

Click to see code generating output

d <-
  file.info(
    list.files(
      path = "/Volumes/davidlucey/aDL/data/yelp_dataset/",
      pattern = ".json",
      full.names = TRUE
    )
  )[c(1, 2)]
data.frame(file = stringr::str_extract(rownames(d), "yelp.*"), size = d$size)

                                               file       size
1 yelp_dataset//yelp_academic_dataset_business.json  152898689
2  yelp_dataset//yelp_academic_dataset_checkin.json  449663480
3   yelp_dataset//yelp_academic_dataset_review.json 6325565224
4      yelp_dataset//yelp_academic_dataset_tip.json  263489322
5     yelp_dataset//yelp_academic_dataset_user.json 3268069927

                                               file       size
1 yelp_dataset//yelp_academic_dataset_business.json  152898689
2  yelp_dataset//yelp_academic_dataset_checkin.json  449663480
3   yelp_dataset//yelp_academic_dataset_review.json 6325565224
4      yelp_dataset//yelp_academic_dataset_tip.json  263489322
5     yelp_dataset//yelp_academic_dataset_user.json 3268069927

Background on Drill

To quote from this guide: Apache Drill - Quick Guide.

Apache Drill is a low latency schema-free query engine for big data. Drill uses a JSON document model internally which allows it to query data of any structure. Drill works with a variety of non-relational data stores, including Hadoop, NoSQL databases (MongoDB, HBase) and cloud storage like Amazon S3, Azure Blob Storage, etc. Users can query the data using a standard SQL and BI Tools, which doesn’t require to create and manage schemas.

We also found the excellent chart shown in SQL on Everything with Apache Drill below on What is Apache Drill and how to setup your Proof-of-Concept

SQL on Everything with Apache Drill

If this could work, it feels like we could fire it up and use it in just about any of our data sources or types. In this post, we are just going to use with a single node as we are only working with one small computer, but it looks like it should be easy to add additional nodes to speed things up.

Sergeant

As usual, none of this would have been possible without an amazing open source package created and shared by a real developer, often in their free time. In this case, we relied on Bob Rudis’ (Drill) {sergeant} package, blog posts and bookdown manual Drill in More than 10 Minutes. He explains that he set up the interface because he saw Drill as a streamlined alternative to SPARK for those not needing the ML components (ie: just needing to query large data sources of disparate types like json, csv, parquet and rdbms). The package allows to connect to Drill via dplyr interface with the src_drill() function, and also the REST API with drill_connection(). Before using {sergeant} though, Java, Drill and Zookeeper must be installed.

Java

Drill requires Oracle JDK 1.8, which is several generations earlier than the version we currently have installed on our Mac. In our first year or two, we tangled with Java because we really wanted to use {tabulizer} to extract tables from pdfs. We burned a lot of time trying to understand the versions and how to install and point to them on Stack Overflow. Just last week, we saw a post looking for advice on loading the {xlsx} package, which depends on Java, as well. One of the magical discoveries we made was Java Environment. Go to Java SE Development Kit 8 Downloads, choose the latest Mac Version of 1.8, and install the .dmg. Then on a Mac, brew install jenv, and it is off to the races. Here we show the Java versions on our machine from the Terminal.

jenv versions

  system
  1.6
  1.6.0.65
  1.8
  1.8.0.261
  14
  14.0
* 14.0.2 (set by /Users/davidlucey/.jenv/version)
  oracle64-1.6.0.65
  oracle64-1.8.0.261
  oracle64-14.0.2

In our first pass, we didn’t understand the different paths, but it doesn’t matter anymore. Just copy/paste the name and put in in the following command and the problem is solved.

jenv global 1.8

And we are good to go, plus we can easily switch back when we are done. It is hard to understate how grateful we are to people who built jenv and brew.

jenv version

1.8 (set by /Users/davidlucey/.jenv/version)

Setting up Apache Drill

The latest version (December 2019) can be downloaded from here, but note with the sale of MapR to Hewlett Packard last year, the project is reported to have been “orphaned”. We took the download and install route, though we subsequently found that using brew install apache-drill might have avoided some of the questions we now have about symlinking (see Zookeeper section below). Apache Drill : Quick Setup and Examples gives step-by-step instructions which might have helped if we had it while installing, but currently have Drill installed in /usr/local/apache-drill-1.1.7.0/ (shown below) though the {sergeant} manual directs to install in the drill/ folder.

ls /usr/local/apache-drill-1.17.0/bin

auto-setup.sh
drill-am.sh
drill-conf
drill-config.sh
drill-embedded
drill-embedded.bat
drill-localhost
drill-on-yarn.sh
drillbit.sh
hadoop-excludes.txt
runbit
sqlline
sqlline.bat
submit_plan
yarn-drillbit.sh

Here there are a few options for running Drill. Running bin/drill-embedded from this this folder, a SQL engine comes up, and queries can be run straight from the command line from a basic UI. We wanted to query from RStudio, so we had another step or two. First, we had to configure the drill-override.conf file in the /conf/ folder above. We followed Bob Rudis’ instructions and named our cluster_id “drillbit1” and zk.connect to our local path as shown below. After these steps, we are able to run and show some sample queries using Drill.

grep "^[^#;]" /usr/local/apache-drill-1.17.0/conf/drill-override.conf

drill.exec: {
  cluster-id: "drillbits1",
  zk.connect: "localhost:2181",  
  store.json.reader.skip_invalid_records: true,
  sys.store.provider.local.path: "/usr/local/apache-drill-1.17.0/conf/storage.conf" 
}

Once this was all in place, the start up to run Drill in the local environment is pretty easy just running bin/drillbit.sh start from in the Terminal. We are not actually running it here in RMarkdown because it froze up the chunk while Drill was running.

# Run in Terminal not in .rmd chunk
~/usr/local/apache-drill-1.17.0/bin/drillbit.sh start

We actually ran it separately in the background from Terminal. Below, we are able to check the status and see that drillbit is running.

/usr/local/apache-drill-1.17.0/bin/drillbit.sh status

/usr/local/apache-drill-1.17.0/bin/drill-config.sh: line 144: let: lineCount=: syntax error: operand expected (error token is "=")
/usr/local/apache-drill-1.17.0/bin/drill-config.sh: line 144: let: lineCount=: syntax error: operand expected (error token is "=")
/usr/local/apache-drill-1.17.0/bin/drill-config.sh: line 144: let: lineCount=: syntax error: operand expected (error token is "=")
/usr/local/apache-drill-1.17.0/bin/drill-config.sh: line 144: let: lineCount=: syntax error: operand expected (error token is "=")
drillbit is running.

The {sergeant} manual also talked about allocating more memory, but we didn’t know how to do this or if it was possible on our small system. There were also other options for setting up a Drill connection, like Docker, so maybe that would help us resolve our issues. It could be that these factors are why we haven’t gotten it to work as well as we hoped.

Zookeeper

There is also the option to run Drill in parallel using Zookeeper discussed in the {sergeant} manual. In the Wiring Up Zookeeper section, it says to have drill in usr/local/drill/ for Mac, and to symlink to the full versioned drill to make it easier to upgrade, but it was vague about this. We noticed that we have a separate folder (~/drill/) in our home directory which has a file udf/ file from the installation, which we understand pertains to “user defined functions” (a subject touched on in Recipe 11 of the {sergeant} manual). We weren’t sure exactly which folder was referred to and reading on Stack Overflow, but we were about three steps away from understanding how this all fit together, so our configuration may not be optimal. When we used Zookeeper with the ODBC connection in parallel instead of “Direct to”Drillbit", if anything, we got slower query times as we will discuss below.

Configuring the Drill Path Storage Plug-in

Drill is connected to data sources via storage plug-ins. The {sergeant} manual mentioned the Drill Web UI passing, but we didn’t realize at first that pulling up localhost:8047 in our browser was an important component for profiling queries. We will show a few of the pages below.

# Run in terminal not .rmd chunk
/usr/local/apache-drill-1.17.0/bin/drill-localhost

In his Yelp blog post, Bob Rudis used “root.dfs” as the path to the Yelp tables. At first, we didn’t understand what this referred to, but it is used as the path to the root of the file system where the data is stored as configured in the storage plug-ins. The “Storage” page of the Drill Web App is in Drill Web App Plug-Ins below. Both his and the Apache documentation also refer the “cp” path to refer to example JAR data in the Drill “classpath”. In addition to the two defaults, all the plug-ins available for hive, mongo, s3, kafka, etc. are also shown below.

Drill Web App Plug-Ins

By clicking on the “Update” button for “dfs”, it is easy to modify the “workspace”, “location” and “defaultInputFormat” with the path to the file with your data as shown in Drill Web App Storage DFS Panel below. In our case, we changed the name of workspace to “root”, the location to “/Volumes/davidlucey/aDL/data/yelp_dataset/” and the defaultInputFormat to “json”. All the different data types are shown further down in “formats”, which is one of the big selling points. According to {sergeant}, it is possible to even combine disparate source types like: json, csv, parquet and rmdbs by modifying formats when configuring “dfs”, while pointing to almost any distributed file system. Once a path is configured in the plug-in, the data in that folder is all set to be queried from RStudio.

Drill Web App Storage DFS Panel

Connecting to Drill via `dplyr`

The first and most basic option to connect given by {sergeant} was via dplyr through the REST API, which was simple using src_drill() mapped to “localhost” port 8047. The resulting object lists the tables, including “dfs.root” workspace, which we configured in the dfs storage page above to point to the folder where we stored the Yelp JSON files. Note that there is no connection object involved with this option, and src_drill() doesn’t offer the option to specify much other than the host, port and user credentials.

db <- src_drill("localhost")
db

src:  DrillConnection
tbls: cp.default, dfs.default, dfs.root, dfs.tmp, information_schema, sys

Here we have loaded the key tables with the tbl() similar to Analyzing the Yelp Academic Dataset w/Drill & sergeant. Note the prefix “dfs.root”, followed by the name of the file from the specified Yelp Academic data set folder surrounded by back ticks. Our understanding is that {sergeant} uses jsonlite::fromJSON() to interact with the files while using the dplyr tbl() method to connect.

Click to see R code to set up check, yelp_biz, users & review tbl()

tic.clearlog()
tic("Loading the four key datasets with: ")
check <- tbl(db, "dfs.root.`yelp_academic_dataset_checkin.json`")
yelp_biz <-
  tbl(db, "dfs.root.`yelp_academic_dataset_business.json`")
users <- tbl(db, "dfs.root.`yelp_academic_dataset_user.json`")
review <- tbl(db, "dfs.root.`yelp_academic_dataset_review.json`")
toc(log = TRUE, quiet = TRUE)
yelp_biz.txt <- tic.log(format = TRUE)

# Source:   table<dfs.root.`yelp_academic_dataset_checkin.json`> [?? x 10]
# Database: DrillConnection
   business_id         date                                                     
   <chr>               <chr>                                                    
 1 --1UhMGODdWsrMastO… 2016-04-26 19:49:16, 2016-08-30 18:36:57, 2016-10-15 02:…
 2 --6MefnULPED_I942V… 2011-06-04 18:22:23, 2011-07-23 23:51:33, 2012-04-15 01:…
 3 --7zmmkVg-IMGaXbuV… 2014-12-29 19:25:50, 2015-01-17 01:49:14, 2015-01-24 20:…
 4 --8LPVSo5i0Oo61X01… 2016-07-08 16:43:30                                      
 5 --9QQLMTbFzLJ_oT-O… 2010-06-26 17:39:07, 2010-08-01 20:06:21, 2010-12-09 21:…
 6 --9e1ONYQuAa-CB_Rr… 2010-02-08 05:56:47, 2010-02-15 04:47:42, 2010-02-22 03:…
 7 --DaPTJW3-tB1vP-Pf… 2012-06-03 17:46:09, 2012-08-04 16:19:52, 2012-08-04 16:…
 8 --DdmeR16TRb3LsjG0… 2012-11-02 21:26:42, 2012-11-02 22:30:43, 2012-11-02 22:…
 9 --EF5N7P70J_UYBTPy… 2018-05-25 19:52:07, 2018-09-18 16:09:44, 2019-10-18 21:…
10 --EX4rRznJrltyn-34… 2010-02-26 17:05:40, 2012-12-29 20:05:04, 2012-12-30 22:…
# … with more rows

# Source:   table<dfs.root.`yelp_academic_dataset_business.json`> [?? x 22]
# Database: DrillConnection
   business_id name  address city  state postal_code latitude longitude stars
   <chr>       <chr> <chr>   <chr> <chr> <chr>          <dbl>     <dbl> <dbl>
 1 f9NumwFMBD… The … 10913 … Corn… NC    28031           35.5     -80.9   3.5
 2 Yzvjg0Sayh… Carl… 8880 E… Scot… AZ    85258           33.6    -112.    5  
 3 XNoUzKckAT… Feli… 3554 R… Mont… QC    H4C 1P4         45.5     -73.6   5  
 4 6OAZjbxqM5… Neva… 1015 S… Nort… NV    89030           36.2    -115.    2.5
 5 51M2Kk903D… USE … 4827 E… Mesa  AZ    85205           33.4    -112.    4.5
 6 cKyLV5oWZJ… Oasi… 1720 W… Gilb… AZ    85233           33.4    -112.    4.5
 7 oiAlXZPIFm… Gree… 6870 S… Las … NV    89118           36.1    -115.    3.5
 8 ScYkbYNkDg… Junc… 6910 E… Mesa  AZ    85209           33.4    -112.    5  
 9 pQeaRpvuho… The … 404 E … Cham… IL    61820           40.1     -88.2   4.5
10 EosRKXIGeS… Xtre… 700 Ki… Toro… ON    M8Z 5G3         43.6     -79.5   3  
# … with more rows, and 5 more variables: review_count <dbl>, is_open <dbl>,
#   attributes <chr>, categories <chr>, hours <chr>

# Source:   table<dfs.root.`yelp_academic_dataset_user.json`> [?? x 30]
# Database: DrillConnection
   user_id name  review_count yelping_since useful funny  cool elite friends
   <chr>   <chr>        <dbl> <chr>          <dbl> <dbl> <dbl> <chr> <chr>  
 1 ntlvfP… Rafa…          553 2007-07-06 0…    628   225   227 ""    oeMvJh…
 2 FOBRPl… Mich…          564 2008-04-28 0…    790   316   400 "200… ly7EnE…
 3 zZUnPe… Mart…           60 2008-08-28 2…    151   125   103 "201… Uwlk0t…
 4 QaELAm… John           206 2008-09-20 0…    233   160    84 "200… iog3Ny…
 5 xvu8G9… Anne           485 2008-08-09 0…   1265   400   512 "200… 3W3ZMS…
 6 z5_82k… Steve          186 2007-02-27 0…    642   192   155 "200… E-fXXm…
 7 ttumcu… Stua…           12 2010-05-12 1…     29     4     6 ""    1pKOc5…
 8 f4_MRN… Jenn…          822 2011-01-17 0…   4127  2446  2878 "201… c-Dja5…
 9 UYACF3… Just…           14 2007-07-24 2…     68    21    34 ""    YwaKGm…
10 QG13XB… Clai…          218 2007-06-04 0…    587   372   426 "200… tnfVwT…
# … with more rows, and 13 more variables: fans <dbl>, average_stars <dbl>,
#   compliment_hot <dbl>, compliment_more <dbl>, compliment_profile <dbl>,
#   compliment_cute <dbl>, compliment_list <dbl>, compliment_note <dbl>,
#   compliment_plain <dbl>, compliment_cool <dbl>, compliment_funny <dbl>,
#   compliment_writer <dbl>, compliment_photos <dbl>

# Source:   table<dfs.root.`yelp_academic_dataset_review.json`> [?? x 17]
# Database: DrillConnection
   review_id   user_id   business_id stars useful funny  cool text        date  
   <chr>       <chr>     <chr>       <dbl>  <dbl> <dbl> <dbl> <chr>       <chr> 
 1 xQY8N_XvtG… OwjRMXRC… -MhfebM0QI…     2      5     0     0 "As someon… 2015-…
 2 UmFMZ8PyXZ… nIJD_7ZX… lbrU8StCq3…     1      1     1     0 "I am actu… 2013-…
 3 LG2ZaYiOgp… V34qejxN… HQl28KMwrE…     5      1     0     0 "I love De… 2015-…
 4 i6g_oA9Yf9… ofKDkJKX… 5JxlZaqCnk…     1      0     0     0 "Dismal, l… 2011-…
 5 6TdNDKywdb… UgMW8bLE… IS4cv902yk…     4      0     0     0 "Oh happy … 2017-…
 6 L2O_INwlrR… 5vD2kmE2… nlxHRv1zXG…     5      2     0     0 "This is d… 2013-…
 7 ZayJ1zWyWg… aq_ZxGHi… Pthe4qk5xh…     5      1     0     0 "Really go… 2015-…
 8 lpFIJYpsvD… dsd-KNYK… FNCJpSn0tL…     5      0     0     0 "Awesome o… 2017-…
 9 JA-xnyHytK… P6apihD4… e_BiI4ej1C…     5      0     0     0 "Most deli… 2015-…
10 z4BCgTkfNt… jOERvhmK… Ws8V970-mQ…     4      3     0     1 "I have be… 2009-…
# … with more rows

[[1]]
[1] "Loading the four key datasets with: : 93.973 sec elapsed"

It takes about two minutes to skim yelp_biz, which seems too long for ~210k rows, and definitely not worth it with the other, much larger files. Analyzing the Yelp Academic Dataset w/Drill & sergeant didn’t give the timing on its queries, but we assume it was much faster than this. The error message recommends that we CAST BIGINT columns to VARCHAR prior to working with them in dplyr, and suggests that we consider using R ODCBC with the MapR ODBC Driver because jsonlite::fromJSON() doesn’t support 64-bit integers. So, we are going to give ODBC a try in the next section and will set up a query to try to take this message into account to see if that makes a difference.

Click to see R code to skim Yelp Business JSON

tic.clearlog()
tic("Time to skim: ")
skim <- skimr::skim(yelp_biz)
toc(log = TRUE, quiet = TRUE)
yelp_biz_skim.txt <- tic.log(format = TRUE)

── Data Summary ────────────────────────
                           Values  
Name                       yelp_biz
Number of rows             209393  
Number of columns          14      
_______________________            
Column type frequency:             
  character                9       
  numeric                  5       
________________________           
Group variables            None    

── Variable type: character ────────────────────────────────────────────────────
  skim_variable n_missing complete_rate   min   max empty n_unique whitespace
1 business_id           0         1        22    22     0   209393          0
2 name                  0         1         0    64     1   157221          0
3 address               0         1         0   118  8679   164423          0
4 city                  0         1         0    43     2     1243          0
5 state                 0         1         2     3     0       37          0
6 postal_code           0         1         0     8   509    18605          0
7 attributes            0         1         2  1542     0    78140          0
8 categories          524         0.997     4   550     0   102494          0
9 hours                 0         1         2   170     0    57641          0

── Variable type: numeric ──────────────────────────────────────────────────────
  skim_variable n_missing complete_rate    mean      sd     p0    p25    p50
1 latitude              0             1  38.6     4.94    21.5   33.6   36.1
2 longitude             0             1 -97.4    16.7   -158.  -112.  -112. 
3 stars                 0             1   3.54    1.02     1      3      3.5
4 review_count          0             1  36.9   123.       3      4      9  
5 is_open               0             1   0.807   0.395    0      1      1  
    p75    p100 hist 
1  43.6    51.3 ▁▂▇▆▂
2 -80.0   -72.8 ▁▁▇▁▇
3   4.5     5   ▁▃▃▇▆
4  27   10129   ▇▁▁▁▁
5   1       1   ▂▁▁▁▇

[[1]]
[1] "Time to skim: : 262.239 sec elapsed"

Setting up and Querying Drill with ODBC

First we had to download and install the MapR Drill ODBC Driver, which wasn’t difficult with the instructions here.

                    name              attribute
6           ODBC Drivers MapR Drill ODBC Driver
7 MapR Drill ODBC Driver            Description
8 MapR Drill ODBC Driver                 Driver
                                           value
6                                      Installed
7                         MapR Drill ODBC Driver
8 /Library/mapr/drill/lib/libdrillodbc_sbu.dylib

Here was our connection using ODBC. Note that “ConnectionType” is specified as “Direct to Drillbit” Wiring Up Drill and R ODBC Style. If we were going with Zookeeper, ConnectionType should be “Zookeeper” and “ZKQuorum” “localhost:2181” instead. Since we have Zookeeper installed, we also tried this, but didn’t notice a big difference. When we run the ODBC connection below, the connection pain in RStudio shows four schemas (“c”, “d”, “i” and “s”), each having no tables.

Click to see R code to connect via ODBC

DBI::dbConnect(
  odbc::odbc(),
  driver = "MapR Drill ODBC Driver",
  Host = "localhost",
  Port = "31010",
  ConnectionType = "Direct to Drillbit",
  AuthenticationType = "No Authentication",
  ZkClusterID = "drillbits1",
  ZkQuorum = ""
) -> drill_con

<OdbcConnection>  Database: DRILL
  Drill Version: 00.00.0000

After setting up the connection, the {sergeant} manual returned a message with the current Drill version, but ours showed a Drill version of “00.00.0000”, so that might be part of to problem. We can see that connecting to the tables with ODBC took almost twice as long as with the dplyr connection, so it seems like we are doing something wrong. When we tried this with Zookeeper (not shown), it took 50 seconds, while 33 seconds with “Direct to Drillbit” (below).

tic.clearlog()
tic("Loading the four key datasets with ODBC: ")
check <-
  tbl(drill_con,
      sql("SELECT * FROM dfs.root.`yelp_academic_dataset_checkin.json`"))
yelp_biz <-
  tbl(drill_con,
      sql(
        "SELECT * FROM dfs.root.`yelp_academic_dataset_business.json`"
      ))
users <-
  tbl(drill_con,
      sql("SELECT * FROM dfs.root.`yelp_academic_dataset_user.json`"))
review <-
  tbl(drill_con,
      sql("SELECT * FROM dfs.root.`yelp_academic_dataset_review.json`"))
toc()

Loading the four key datasets with ODBC: : 133.307 sec elapsed

tic.clearlog()

The skim() for yelp_biz took about the same amount of time, but either way, it was still way too long to be a viable alternative. Again, “Direct to Drillbit” here took 116 seconds, while 81 seconds with Zookeeper, so we are clearly doing something wrong if all the things which are supposed to speed things up are actually slowing us down.

Click to see code to skim Yelp Business JSON with ODBC

tic("Skim yelp-biz with ODBC")
skim1 <- skimr::skim(yelp_biz)
toc(log = TRUE, quiet = TRUE)
yelp_odbc_skim.txt <- tic.log(format = TRUE)

Table 1: Data summary
Name	yelp_biz
Number of rows	209393
Number of columns	14
_______________________
Column type frequency:
character	11
numeric	3
________________________
Group variables	None

Variable type: character

skim_variable	n_missing	complete_rate	min	max	empty	n_unique
business_id	0	1	22	22	0	209393
name	0	1	0	64	1	157229
address	0	1	0	118	8679	164423
city	0	1	0	43	2	1251
state	0	1	2	3	0	37
postal_code	0	1	0	8	509	18605
review_count	0	1	18	21	0	1320
is_open	0	1	1	21	0	2
attributes	0	1	3	1713	0	78140
categories	524	1	4	550	0	102494
hours	0	1	3	206	0	57641

Variable type: numeric

skim_variable	complete_rate	mean	sd	p0	p25	p50	p75	p100	hist
latitude	1	38.58	4.94	21.50	33.64	36.15	43.61	51.30	▁▂▇▆▂
longitude	1	-97.39	16.72	-158.03	-112.27	-111.74	-79.97	-72.81	▁▁▇▁▇
stars	1	3.54	1.02	1.00	3.00	3.50	4.50	5.00	▁▃▃▇▆

[[1]]
[1] "Skim yelp-biz with ODBC: 115.719 sec elapsed"

Drill Web App

As we go along making queries, everything is collected in the Web App Profiles page, as shown in Drill Web App Query Profiles just below. Clicking on a query here takes us to the Query and Planning page, shown in further down in Drill Query and Planning Pane for Complicated SQL Query. There are other dashboards which we will show below.

Drill Web App Query Profiles

Query Profiling with Drill

The other interesting thing in Drill was profiling. Here is a more complicated query we experimented with with a couple of joins and some aggregations for a query which wound up taking over an hour. See that we CAST the integer variables in this case as we were warned above, but that also didn’t seem to make a difference.

dq <-
  odbc::dbGetQuery(drill_con, 
    "SELECT b1.name
            ,CAST(b1.stars AS INT) AS stars
            ,CAST(b1.review_count AS INT) AS review_count
            ,c.reviews
      FROM (SELECT b.business_id
              ,COUNT(*) as reviews
      FROM dfs.root.`yelp_academic_dataset_user.json` AS u,
            dfs.root.`yelp_academic_dataset_review.json` AS r,
            dfs.root.`yelp_academic_dataset_business.json` AS b
      WHERE r.user_id = u.user_id
            AND b.business_id = r.business_id
      GROUP BY b.business_id, r.user_id
      HAVING COUNT(*) > 10) AS c
      INNER JOIN dfs.root.`yelp_academic_dataset_business.json` b1
      ON c.business_id = b1.business_id"
      )

We are not running the query here in the blog post, but as mentioned, the timing can be seen in Drill Query and Planning Pane for Complicated SQL Query below the query at 1h11.

Drill Query and Planning Pane for Complicated SQL Query

It is amazing how much information about the query Drill gives us, shown in Drill Query and Planning Pane for Complicated SQL Query above Clicking on the “Edit Query” tab, and scrolling down to Operator Profiles (shown below), we can see that we some operators spilled to disc and that the scan operators spent more time waiting for data than processing it. We can also see that the Hash Aggregate in Fragment 1 took 13% of the query time. Further down but not shown, the Hash Joins took almost 70% of the query time, so the Hash Joins and Hash Aggregate together took 70% of the query time. Even without those bottlenecks, we probably still wouldn’t have been satisfied with the amount of time this took. Having this information, it seems like it would be possible to optimize, but we didn’t know how to do it. We have been recently learning SQL and realize that there is still a lot to learn.

Drill Operator Profiles for Complicated SQL Query

Lastly, Drill has a nice dashboard which allowed us to for example instruct the hash joins and hash aggregations to ignore memory limits as shown in Drill Web App - Options Panel below. There were a lot of parameter settings available, but we were not sure how to adjust these to solve our specific problems, but would welcome any good advice or pointers.

Drill Web App - Options Panel

Clean up

Shutting down when done is also easy as shown here.

/usr/local/apache-drill-1.17.0/bin/drillbit.sh stop

Returning to JDK 14.0

jenv global 14.0.2

Conclusion

We don’t know the status of Drill given “orphan” status, but there wasn’t much current discussion that we could find with a quick search. If these problems are fixable, we would be very grateful for feedback and promise to update this post for the benefit of others. We have read that the arrow package is a lot faster than this on similar sized data, but don’t know if it is as flexible. If there is a clearly better open source way to accomplish these objectives, such as arrow, any guidance would be much appreciated.

Finding the Dimensions of `secdatabase.com` from 2010-2020 - Part 2

Mon, 12 Oct 2020 00:00:00 +0000

Source: xbrl.org

Introduction

This is part 2 of a 3-part series on extracting XBRL data from secdatabase.com. In Learning SQL and Exploring XBRL with secdatabase.com - Part 1, we showed how to set up the database connection from RStudio. In this post, we will discuss the basics of XBRL and secdatabase.com’s SQL representation of it. We will try to understand the dimensions of the database over its history, and how to query it from RStudio. For example, how many unique companies per year by filing type are included? How many unique labels and concepts per company each year? Have report names been standardized so that data can be retrieved and compared across sectors and over time? How many revisions and extensions were used over time? The goal is to explore how the project has progressed since launch from the perspective of usability for financial analysis in its raw form.

In the first few years, there were several academic papers probing the usefulness of XBRL for end investors, but we found fewer more recently. In this analysis, we find that some of those concerns are starting to be resolved, but many remain valid today. We would like to forewarn that we are not experts on XBRL, and are making our best effort to understand this complicated data set. XBRL is unstructured, and may or may not have consistent fields, while SQL is by definition structured, so this adds complexity. In addition, most of the resources available about XBRL are for accounting professionals, and few that we could find, went into depth about financial analysis. The purpose of our blog is to record our learning so that we can refer back, and also to get feedback from others if there are better solutions.

We would also like to warn in advance that this post will include a lot of cod on how to query secdatabase.com with SQL as well as some quite detailed discussion of the XBRL data model, though we used R for charts and tables. It is intended to be helpful for readers who really want to better understand the history and structure XBRL, and possibly to try out secdatabase.com for themselves. Now would be a good time to drop off or skim through the tables if that will be too much detail. Lastly, all the queries shown were checked by secdatabase.com, who were very welcoming of the project. There is a comment section at the bottom, and we welcome any constructive feedback.

Series Index

Part 1 - Learning SQL and Exploring XBRL with secdatabase.com
Part 2 - Finding the Dimensions of secdatabase.com from 2010-2018 - Part 2
Part 3 - How to extract company or sector line items from secdatabase.com - Part 3 (In Progress)

Hierarchial Taxonomy of XBRL in `secdatabase.com`

From an investor’s perspective, the most challenging thing about XBRL is learning the taxonomy to be able to extract the desired financial statement elements from deeply nested structures. We are going to do our best to lay it out succinctly, but it is likely that there will be inaccuracies. We relied on The XBRL Book for much of the information about XBRL, and a lot of responses from the secdatabase.com team about their database. The XBRL Book argues that everything we find to be complicated about XBRL are misconceptions, but after our deep dive, we respectfully disagree.

The root of XBRL data elements are called facts, which are recorded as string or numeric values, represented by a series of aspects (concept, period, entity and unit) within a context. Within secdatabase.com, facts are stored in three “data_point” tables, with the most important being “data_point_snapshot” (which shows the most current value after any revisions). As we showed in the last posts, all the “entities” (identified by “cik” or “company_name” fields in secdatabase.com) and all their filings (ie: 10-K’s, etc.) are tracked in the “company_submission” table. The combination of two or more aspects, called a “collision”, can uniquely specify the coordinates of a particular value or group of values (facts) within a context to be displayed in the presentation of a particular entity. Within secdatabase.com, this always involves joining two or more tables together, usually on at least the accession_number_int field.

Aside from the entity (“company_name” or “cik” in secdatabase), probably the most important aspect is the concept (such as “Revenues”, “Assets”, etc.), but without other aspects, no value can be uniquely specified. Instead of calling these concepts, secdatabase.com uses the field “datapoint_name”, which is included in three tables having the term “data_point” in them, as well as the “report_presentation_line_item” table. We have also heard concepts called “tags” in some cases. A particular “datapoint_name” is likely to have many “datapoint_ids” attached to it. For example, a single 10-K might have three separate facts specified by a “datapoint_name” like “Revenues” (for say 2017, 2018 and 2019), but each of these would have a unique “datapoint_id”. If the entity or year was not specified, a single “datapoint_name” might bring thousands of facts for “Revenues” (ie: for all the companies in that year). We found it complicated to specify the context to get the unique value we were hoping for, and often found ourselves with duplicates for some companies and none for others. We will discuss this more in Post 3 where we try to use the data for company or sector analysis.

An XBRL Instance is just a “bag of facts”, and a single fact might be shown in more than one “presentation” (statements and disclosures). For example, Total Revenues might be displayed in the Income Statement, but also in a disclosure of segment reporting of a region or product. A separate taxonomy schema defining concepts and a linkbase to organize specified concepts into graphs to be displayed in presentations. We discovered that the names used to describe the same element varied among companies, and the same company might even change the names and/or formatting of its own financial statements over time (ie: “statement - BALANCE SHEET” or “statement - Balance Sheet”), making them different to a computer. As a result, secdatabase has manufactured a identifier in the “report_presentation_section” called the “statement_type”, which tries to collapse these many variations into the main financial statements which we all expect (ie: Balance Sheet (B), Income Statement (I), etc.). The believe their field is right most of the time, but we found even this identifier to be an incomplete solution.

When a fact has been recorded against a concept (or “datapoint_name”) as a “string_value” or “numeric_value”, XBRL also assigns “label roles”, which can be used ultimately be used to specify how it will be displayed in a presentation (it’s concept or in secdatabase.com speak “database_name” is not displayed in the presentation). Every label has a “language” and a “label role” (called “preferred_label_role” by secdatabase.com), and like concepts, labels are part of a linkbase. A single concept may be associated with more than one label. If this seems confusing, that is because it is.

Distinct Companies and Filings per Annum Have Been Declining

We wanted to understand the universe of companies which have been filing in XBRL each year, which was achieved by a simple SQL query of the single “company_submission” table. In Figure 1 below, we group by “document_type” and “document_fiscal_year_focus”and eliminate duplicate filings with “DISTINCT”, then we count by year and “document_type”. We had heard that there are ~ 8,000 companies filing in XBRL, so adding up the 10-K, 10-K/A and 20-F’s, we get close to that many in 2012, but then easing off to ~ 6,600 in recent years. We assume companies have been withdrawing listings or being acquired at a faster rate than new companies have been issuing stock. Though there was a healthy increase in foreign companies filing 20-F’s, it is surprising to see a decline in equity listings during an historic bull market.

SELECT document_fiscal_year_focus AS year
      ,document_type AS filing
      ,COUNT(DISTINCT accession_number_int) AS num_filings
      ,COUNT(DISTINCT cik) AS num_companies
FROM sec_financial_statements.company_submission
WHERE document_fiscal_year_focus BETWEEN 2010 AND 2020
      AND document_type in ('10-K', '10-K/A', '10-Q', '10-Q/A', '20-F', '20-F/A', '40-F')
GROUP BY document_fiscal_year_focus, document_type
ORDER BY document_fiscal_year_focus, filing;

Figure 1: Number of Edgar Filings by Type and Year

History of XBRL Challenges

In the early days after launch of XBRL, there was lot of academic research on the challenges of implementing XBRL. Here is a thorough accounting of the concerns from the beginning of 2013 SEC–Mandated XBRL Data at Risk of Being Irrelevant to Investors and Analysts. The key recommendations to avoid the risk of becoming obsolete to investors were: (1) reduce the error rate and limit unnecessary extensions, (2) improve the quality of data and (3) have technologists take over from accountants. The link to CEASA’s longer paper on the subject is here. As we will show, between vendors like XBRLogic and the industry advocate, XBRL US, the US branch of the non-profit supporting the implementation of digital business reporting standards, there are signs of recent improvement on some fronts.

Labels Role Types Contribute to Errors

One of the big contributors to presentation errors comes from labels. In the next query, we show the number of label roles each year, this time from the “report_presentation_line_item” table, which connects a group of concepts to a presentation. By far the most common is the terseLabel, followed by no or a missing label, verboseLabel and label (also known as the “standard” label). We also don’t understand why there would not be a label in so many cases. As we understand it, the terseLabel and verboseLabels are shorter and longer versions of label, respectively. We are surprised that there are so many more terseLabels than labels or verboseLabels, because the XBRL Book informs us that the standard label is supposed to be the most common. Both labels and terseLabels are then aggregated within the XBRL Taxonomy (which we showed at the end of Part 1) into totalLabels, which are always sums of other labels. The periodEndLabel and periodStartLabel are used to define the period or duration of time, and are used with other to define period. In the query below, we pull out the most common “preferred_label_role” over the whole period.

SELECT preferred_label_role
       ,COUNT(*) AS frequency
FROM sec_financial_statements.report_presentation_line_item
GROUP BY preferred_label_role
ORDER BY COUNT(*) DESC;

Figure 2: Frequency of Usage of Label Roles - All Periods

Because facts are supposed to be positive by default, there are several “negated” label roles, which are used if there is a need to flip the sign of something for display in a presentation. For example, the main concept for net income is “NetIncomeLoss”, which should by convention always have a positive numeric value, but would be displayed with a negatedLabel when there was a loss. However, we also found cases where the string or numeric value was already negative. Our understanding is that one of the most common areas where errors occur in XBRL is with negated labels mistakenly transposing the sign of line item. With all of this complexity, it is not surprising that the wrong labels or label roles have often been used. As we will discuss, label roles seem to be one area where there confusion by those who prepare XBRL statements, seemingly switching conventions from year to year. If those who prepare financial statements have struggled with it, the challenge would be doubly difficult for investors. After we spent several hours trying to understand label roles, we are still not sure we have it down.

The Problem of Concept Standardization

The complication surrounding concepts (in our data set called “datapoint_name”), seems to be another of the main challenges which has held XBRL back from wide use by investors. Companies have the discretion to make up custom concepts, known as “extensions” in XBRL parlance. The desire to do this company-by-company makes sense, but when considered across sectors or the market as a whole, it negatively impacts comparability. All of these issues are discussed in detail in this excellent summary by Idaciti In Data We Trust. In High Quality Data We Shine.

In Figure 3, the 25 most common fields in 2019 are shown. Our understanding is that XBRL is built on the idea of matching “as reported” financial statements. Though it seems like the highest level line items like “Assets”, “Liabilities” and “Shareholders Equity” should have 100% comparability across the 5,460 companies reporting 10-K’s in 2019, these come well short of the number. The number of companies using a field also falls off sharply after the first 25, so comparability will be much worse for more revealing items, like bad debts, capitalized items and accruals.

SELECT datapoints
      ,COUNT(*) AS num_occur
    FROM (SELECT DISTINCT dp.datapoint_name AS datapoints, 
                 dp.accession_number_int,
                    document_fiscal_year_focus
    FROM sec_financial_statements.data_point_snapshot dp
    INNER JOIN sec_financial_statements.company_submission cp
      ON dp.accession_number_int = cp.accession_number_int
    WHERE document_fiscal_year_focus = 2019 
      AND dp.unit = 'USD' 
      AND cp.document_type = '10-K')
GROUP BY datapoints
ORDER BY num_occur DESC
LIMIT 25;

Figure 3: Number of Occurances of Concept during 2019

In the query below, with results shown in Figure 4, we show the number of unique concepts (datapoint_name) by “label_role” over time. The number of unique concepts rise steadily until 2013 or 2014, and then start to come down slowly. The number of labels roles which are missing (ie: no label role at all recorded) shows the most improvement. So there is some evidence that an effort is being made, but as we will show in Part 3, there is still room to improve. The only thing to say about the SQL query are to highlight the two WHERE statements (1) rpli.datapoint_id = dp.datapoint_id and (2) rps.section_sequence_id = rpli.section_sequence_id. Without these two constraints, we found we were double-counting many data points.

SELECT year
      ,label_role
      ,COUNT(*) AS unique_labels
FROM( SELECT DISTINCT rpli.datapoint_name
                      ,rpli.preferred_label_role AS label_role
                      ,cs.document_fiscal_year_focus AS year
      FROM sec_financial_statements.company_submission cs,
            sec_financial_statements.report_presentation_section rps,
            sec_financial_statements.report_presentation_line_item rpli,
            sec_financial_statements.data_point_snapshot dp
      WHERE cs.accession_number_int = rpli.accession_number_int 
        AND rpli.accession_number_int = rps.accession_number_int 
        AND rps.accession_number_int = dp.accession_number_int 
        AND rps.section_sequence_id  = rpli.section_sequence_id 
        AND rpli.datapoint_id = dp.datapoint_id 
        AND cs.document_fiscal_year_focus BETWEEN 2010 AND 2020 
        AND cs.document_type = '10-K' 
        AND cs.document_fiscal_period_focus = 'FY'
      GROUP BY cs.document_fiscal_year_focus, rpli.datapoint_name, rpli.preferred_label_role)
GROUP BY year, label_role
ORDER BY year;

Figure 4: Annual Number of Unique Concepts by Label

Use of Extensions Has Declined, But Still Represent About 18% of Data Points

It is easy to understand why extensions are needed, but non-standard data points are big detractors from comparability, so it is recognized for a long time that fewer would be better. It sounds like the SEC is getting more serious more recently about discouraging them, but progress has been slow, especially for smaller companies. With the help of secdatabase.com, the query below filters on the “version” from the “data_point” table being null or not null in order to distinguish non-standard from standard data points. If version (ie: “us-gaap/2019”) is null, the data point is an extension. In our query, we have limited to just 10-K’s and 10-K/A’s.

We can see that almost all companies have at least a few extensions. Our understanding is that larger companies have been much better at sticking to the standard tags, but we have not grouped by company size for this analysis. The results of the query in Figure 5 shows that extensions peaked in about 2013, and have come down by almost half. We do also not show the calculations here, but there are about 30x as many unique non-standard as unique standard data point names, but because standard data point names are used so much more frequently, the ratio of non-standard to standard data points is much lower. As we show below, we calculate that the ratio has bounced between 16-25%, and in recent years, extensions have been about 18% of all data points. This ratio squares with with the SEC GAAP Trends Analysis for 2019.

SELECT year(cs.filing_date) AS filing_year
   ,count(CASE WHEN version IS NOT NULL THEN datapoint_name END) standard_datapoints_count
   ,count(CASE WHEN version IS NULL THEN datapoint_name END) non_standard_datapoints_name_count
   ,count(DISTINCT CASE WHEN version IS NULL THEN cs.cik END) total_company_count_with_extension
   ,count(DISTINCT cs.cik) total_company_count
FROM sec_financial_statements.data_point_snapshot dp
INNER JOIN sec_financial_statements.company_submission cs
   ON dp.accession_number_int = cs.accession_number_int
WHERE CS.document_type IN ('10-K', '10-K/A') AND dp.segment_hash IS NULL
GROUP BY year(cs.filing_date)
ORDER BY 1 DESC

Figure 5: Standard Data Points and Extensions Over Time

Another Way to Look at Annual Number of Unique Tags

For the query below shown in Figure 6, we sum up the annual number of times the five most common label roles (label, terseLabel, verboseLabel, totalLabel and no label) were used for a unique concept (datapoint_name). We then summed up how many concepts were used that many times in a given year with that label role. If the “Concept Frequency in Year” (x-axis) is 1, each of those label roles were only used once for that year, 2 for those used twice, and so on. Taking for example the label facet in the chart below, the number used just 1x has generally been ~4,000-10,000 per year. Unique “terseLabel’s” were used 1x ~68,000-90,000. If there are typically 6-8k companies filing, there should ideally be only several hundred concepts used that many times for genuine comparability. Moving to the right of the x-axis, there are very few labels which were used more than 500 times. Curves for later years have come down somewhat and slowly flattened compared to the early years. Positive change would see the graph moving much further downwards from the left on the tag_frequency axis and flattening out.

SELECT year
        ,tag_frequency
        ,label_role
        ,COUNT(*) AS num_repeats
FROM( SELECT DISTINCT rpli.datapoint_name
                     ,rpli.preferred_label_role AS label_role
                     ,COUNT(*) as tag_frequency
                    ,cs.document_fiscal_year_focus AS year
      FROM sec_financial_statements.company_submission cs,
           sec_financial_statements.report_presentation_line_item rpli,
           sec_financial_statements.report_presentation_section rps,
           sec_financial_statements.data_point_snapshot dp
      WHERE cs.accession_number_int = rpli.accession_number_int 
        AND rpli.accession_number_int = rps.accession_number_int 
        AND rps.accession_number_int = dp.accession_number_int 
        AND rps.section_sequence_id  = rpli.section_sequence_id 
        AND cs.document_fiscal_year_focus BETWEEN 2010 AND 2020 
        AND rpli.datapoint_id = dp.datapoint_id 
        AND cs.document_type = '10-K' 
        AND cs.document_fiscal_period_focus = 'FY' 
      GROUP BY cs.document_fiscal_year_focus, rpli.datapoint_name, rpli.preferred_label_role)
GROUP BY year, tag_frequency, label_role
ORDER BY year;

Figure 6: Annual Concept Repeats by Concept Frequency of Occurance

It would be more understandable a year or two after the launch of XBRL, but when all the warning signs were there from the beginning, is a little frustrating to still see. The Idaciti’s In Data We Trust. In High Quality Data We Shine does an excellent job of explaining these comparability issues. As, mentioned earlier describes recent efforts being made to improve this situation by the FASB, the XBRL US Data Quality Committee and the SEC, which we will discuss later on.

In response to these challenges, Intrinio, an XBRL and market data provider, uses machine learning to “standardize” 18,000 unique concepts (they use the term “tags”) each year down to about 300, which allows for better comparability. They discuss their standardization process here How Does Intrinio Build Standardized Fundamental Data?. Intrinio reports that their algorithms classify financial statement fields with 99.8% accuracy before human review. We have heard others, like Compustat, do much of this manually. Absent building our own standardization tools, these services may be the way to go for the time being.

Many Different Names for the Same Financial Statements

In Figure 7, we show the most used names for the various financial statements in 10-K’s for 2019. For example, there are a dozen different names for the Consolidated Balance Sheet with the most common, “Statement - Consolidated Balance Sheets”, being shared by less half of the reports filed that year. Some are capitalized, others leave out the word “Consolidated” and “Financial Condition” is used instead of “Balance Sheet”. There may be some issues also with the secdatabase.com categorization because “Condensed” rather than “Consolidated” are also included here in some cases. Ideally, there would be only one tag for each of these reports for all reporting companies. The other statements have even more variations than for the Balance Sheet. As we will discuss in Part 3, this presents problems when trying to query a single financial statement line item from secdatabase.com from a group of companies or time periods.

SELECT rps.report_section_description AS report_name
      ,rps.statement_type
      ,COUNT(*) AS num_reports
FROM sec_financial_statements.report_presentation_section rps,
      sec_financial_statements.company_submission cs
WHERE rps.accession_number_int = cs.accession_number_int 
  AND cs.document_type = '10-K' 
  AND cs.document_fiscal_period_focus = 'FY' 
  AND cs.document_fiscal_year_focus = 2019 
  AND rps.statement_type IN ('I', 'B', 'C', 'SE')
GROUP BY rps.report_section_description, rps.statement_type
ORDER BY COUNT(*) DESC
LIMIT 100;

Figure 7: Actual Report Name by Financial Statement Type in 2019

`secdatabase.com` Distinct Statement Type Counts

Because of the varied names used for the same statements, secdatabase.com created the “statement_type” field as a classifier. The actual disclosure name can be found in the “report_section_description” variable of the “report_presentation_section” table. In the query below with results shown in Figure 8, we aggregated those by year. There are a similar number of Balance Sheets (“B”), Statement of Shareholder’s Equity (“SE”) and Statements of Cash Flows (“C”), but much fewer Income Statements (“I”) than there are 10-K’s. There are a very large number of unique names for Disclosures. Many more would have been expected, but about 60 Disclosures per company is more than we would have expected. We can see that they come close to the 6,600 Balance Sheet, Statement of Cash Flows and Statements of Stockholders Equity in 2019, but fall short on Income Statements.

SELECT statement_type AS statement_name
      ,document_fiscal_year_focus AS year
      ,COUNT(*) AS frequency
FROM sec_financial_statements.report_presentation_section rps,
      sec_financial_statements.company_submission cs
WHERE cs.accession_number_int = rps.accession_number_int 
  AND cs.document_type IN ('10-K', '10-K/A', '10-Q', '10-Q/A', '20-F', '20-F/A', '40-F')
  AND cs.document_fiscal_period_focus = 'FY'
GROUP BY statement_type, document_fiscal_year_focus
ORDER BY COUNT(*) DESC;

Figure 8: Annual Financial Statement Type by Frequency

XBRL Reporting Errors Past and Present

As mentioned earlier, there can also be many kinds of errors including taxonomy errors, block type, sign errors, extensions, period type and others. We had hoped to try to replicate the excellent quarterly XBRL Quality Score shown below, but secdatabase.com only parses the database without validation, so it is impossible at least for now. The table is far beyond what we could hope to achieve and errors across 9 metrics along with their prevalence. In their blog post THE XBRL FILES: OF WHAT USE ARE THE RULES?, XBRLogic expresses many of the same frustrations that we have discussed above.

Source: XBRLogic

On its website, XBRL US has built an excellent Filing and Results and Quality Checks Dashboard, which compares the Data Quality Committee and SEC Edgar Filing Manual rules to company as reported filings. There are also tools put in place starting in 2015 for companies to check for errors before filing. They have also tracked the progress over the most common errors over time in the Aggregated Real-time Filing Errors report. In the initial years, many of the errors kept rising, but in the last two years have come down sharply. Although we don’t have the validation data to find the errors, we assume that most errors eventually end up as revisions, which we do have. With the help of secdatabase.com, we were able to put together the query below. If the results in Figure 9 are anything to go by, there was a slow decline in recent years, followed by a big drop in 2019. We can hope that such excellent feedback like XBRLogic table and the XBRL US tools are leading to the sharp recent reduction in errors.

Figure 9: Annual Revision Count over Time

Conclusion

Judging by how little academic research, books or other blog posts we could find in more recent years (outside of those by data vendors), it seems like CEASA’s concerns may have at least partially come to pass with the raw XBRL data. It seems like errors are becoming much less common and may go away completely in future statements, though unfortunately, are likely to remain part of the historic record. We mentioned earlier that Intrinio has been using machine learning reportedly with success to “standardize” XBRL elements, and that deservedly has a cost. Maybe vendors will have to be a layer between the raw data and end investors, but they often have user interfaces which lead to constraints and slow down access relative to a big database like this one. When we queried secdatabase.com, no response took more than a few seconds, no matter how many rows, which may not be possible with other vendors, and having that access may not be affordable or allowed for boundless exploration as we have done here. A future series could involve exploring if a vendor like Intrinio would be a viable means for accessing data points across the full historical record of companies and years in a cost effective manner. It is easy to get excited about the potential of XBRL to mine for insight, but this exercise has been another reminder that it is still a work in progress. In our next post, we will try to drill down by company and across sectors to see how we do.

Redwall Analytics

Amateur Urbanist Critique of Work-Live-Ride using 360k Fairfield County Parcels

Introduction

Collecting, Loading and Cleaning Data

Code for Filtering and Aggregating Parcels within Ranges

Results and Analysis

Perspective on Greenwich Topology and WLR Impact

Final Thoughts

NYED Data Explorer Shows 15 Years of Charter School Success

Introduction

Classical Charter Schools and the Charter Debate

Thoughts on NYED and Open Public Data

Data Collection Considerations

Navigating the App

Future additions

Conclusion

Handling larger than memory data with {arrow} and {duckdb}

Introduction

Other Resources

Loading an {arrow} Table

Loading and Preparing the Bad Data to Stack

Test Query and Benchmarking

Conclusion

Loading a large, messy csv using data.table fread with cli tools

Introduction

Tools and Setup

The Problem

Using csvclean

Bad Rows

Further Explorations

Conclusion

Exploring Stock Market Listing Mortality since 1986

Introduction

Potential for Survivor Analysis with Sharadar Coverage

Exploring Security Life and Death with Sharadar’s Coverage

Coding up Survivor Analysis using the Magic of {data.table}

Building the Model and Plotly Object

Thoughts on Survival Rates by Period

Conclusion

When Yahoo Finance doesn't have de-listed tickers needed

Introduction

Financial Data API’s for De-Listed Securities

Finding Matching Prices

Some thoughts on the collected data

Conclusion

Introducing the Redwall 'Red Flag' Explorer with New Constructs Data

Key Findings

Introduction

Red Flag Calculations

Thoughts on Red Flags

Ex-Post Return Data

Bias From Missing Return Data

Using the Shiny App

Cursory Analysis

Thoughts about Data and Modeling

Next Steps

Conclusion

Introducing the Redwall IRS SOI Tax Dashboard

Introduction

Methodology

Data Summary

Conditional Drop Down Filtering

Using the Shiny App

Assessing the App

Observations

Conclusion

A Blueprint of "Red Flag" alerts Using Adjusted Earnings Data

Introduction

New Constructs

The Forgotten Art of Forensic Accounting

Key Accounting Diseases

CASH FLOW Quality

EARNINGS QUALITY

A/R & Inventory (Balance Sheet)

Accruals (Balance Sheet)

Earnings Quality (Earnings Quality)

EFFICIENCY

Returns (Efficiency)

Margins & Return Trends (Efficiency)

Turnover (Efficiency)

Perspective on Greenwich Topology and `WLR` Impact

Loading an `{arrow}` Table

Using `csvclean`

Coding up Survivor Analysis using the Magic of `{data.table}`

The Willamette Government Finance Database and `{govFin}` Package

The `{SQLite}` Database

Connecting to Drill via `dplyr`

Hierarchial Taxonomy of XBRL in `secdatabase.com`

`secdatabase.com` Distinct Statement Type Counts