Training Sample

The sampling frame for our training sample are all US towns with a population greater than 1000, as measured by the 2010 census. Specifically, we use census data on “incorporated places” as our target population, which amounts to 39548 administrative units.

Our training sample consists of manually labeled indicators from 818 towns. The sample is broadly representative of the population of towns with populations greater than 1000 in the United States, with some over-representation of particular states such as New York and Texas.¹ Because of the over-representation of towns from a few states, the typical town in our training sample is larger in population than the typical town in the US, but the distributions overlap substantially. This comparison is visualized in Figure 1.

Figure 1: Distribution of Town Population in Training Sample and All US Towns with population greather than 1000.

The training sample towns represent 49 distinct states. The geographic distribution of the training sample by state is depicted in Figure 2.

Figure 2: Geographic Distribution of Towns in the Training Sample

Feature Selection

The feature selection step relies on calculating a bivariate correlation between the transparency indicator and each column of the document term matrix. Specifically, we compute the information gain between the indicator and each column of the document term matrix, which is a univariate measure of association between the two variables.. We then rank the columns with respect to information gain and keep the top \(p\)% percent of the variables, where \(p\) is chosen via cross-validation.

The proportion of terms retained for each model are as follows:

• AGD: 0.11
• BDG: 0.57
• BID: 0.27
• CAFR: 0.13
• MIN: 0.2
• REC: 0.12

Random Forest Model

After weakly correlated variables are pruned from the document term matrix via the feature selection step, we use a Random Forest model to predict the transparency indicator. Specifically, we use the implementation in the ranger(Wright and Ziegler 2017) package, which is optimized for speed. Each model is an ensemble of 500 different classification trees. When constructing individual trees, Random Forest models use only a random subset of both the training sample and the predictors (post-feature selection). Specifically, each tree uses a random sample (with replacement) of 0.9 of the number of units in the training sample. The predictors used in each tree are also a random sample, where the number of predictors retained is equal to the square root of the total number of predictors in the post-feature selection DTM.

The only model hyper-parameter that is tuned performed is the proportion of variables to retain via the feature selection step described above. We use cross-validation on the combined feature selection and random forest procedures to choose an optimal proportion of variables to retain.

To understand what aspects of the website are used when predicting each label, we computed the permutation importance of each column of the DTM. Specifically, we permute each column of the DTM and calculate the increase in prediction error. This statistic provides one measure of the relative importance of each unigram count from the website. The results for the 6 models can be found in Figure 3. Unsurprisingly, the most predictive unigrams tend to be the labels themselves, such as “budgets”, “minutes”, etc.

Figure 3: Permutation Importance. Shows 10 columns of the document term matrix that have the highest permutation importance.

Out of Sample Performance

To estimate performance of the model, we use cross-validation. Specifically, we do 5-fold cross-validation, repeated 10 times. Table 1 reports the results for overall accuracy, positive predictive value, and negative predictive value. Positive predictive value is defined as the proportion of correct labels among sites labeled as having the indicator, while negative predictive value is the proportion of correct labels among sites labeled as not having the indicator.