Results

Model Selection

            Call:
            glm(formula = isBot ~ rate, family = "binomial", data = training_data)

            Coefficients:
                    Estimate Std. Error z value Pr(>|z|)
            (Intercept) 0.003402   0.098073   0.035    0.972
            rate        0.214539   0.202490   1.060    0.289

            (Dispersion parameter for binomial family taken to be 1)

            Null deviance: 775.87  on 559  degrees of freedom
            Residual deviance: 774.61  on 558  degrees of freedom
            AIC: 778.61

            Number of Fisher Scoring iterations: 4

        

            Call:
            glm(formula = isBot ~ rate + sd_inter_response, family = "binomial", 
                data = training_data)

            Coefficients:
                   Estimate Std. Error z value Pr(>|z|)    
            (Intercept)     18.8576     1.6917  11.147   <2e-16 ***
            rate            -7.9457     0.9557  -8.314   <2e-16 ***
            sd_inter_response  -1.6746     0.1563 -10.711   <2e-16 ***
            ---
            Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

            (Dispersion parameter for binomial family taken to be 1)

            Null deviance: 775.87  on 559  degrees of freedom
            Residual deviance: 245.69  on 557  degrees of freedom
            AIC: 251.69

            Number of Fisher Scoring iterations: 7

        

In the first model, rate is not statistically significant (p = 0.289). However, when sd_inter_response is included as a second predictor, both rate and sd_inter_response become highly significant (p < 0.001). Additionally, the second model's AIC (251.69) is substantially lower than the first model's (778.61), indicating a much better fit. This suggests that the combination of response rate and variability in response timing provides stronger predictive power than response rate alone. Thus, our final model is:

\[p = \frac{e^{-7.95\cdot\text{rate} - 1.67\cdot\text{sd_inter_response}}}{1 + e^{-7.95\cdot\text{rate} - 1.67\cdot\text{sd_inter_response}}}\]

This is just one model built using one method among many possible options. Before using it to support forecasting or decision-making, we need to evaluate how well it generalizes to new data. This logistic model was trained on one portion of the data; we now assess its performance on an independent test set. While more advanced techniques—such as k-fold cross-validation—can be used during training to improve robustness, we do not cover those in this project.

ROC Curve

Using the description above, with an AUC of 0.94, we can say that 94% of the time, when we randomly select one bot and one human case, the model assigns a higher score to the bot. But what counts as a “good” AUC depends a lot on what you're modeling. Take recidivism: regular recidivism is difficult to predict, and violent recidivism is more difficult. So an AUC of 0.6 in that context might be acceptable, depending on what's at risk. But for something like human vs. bot response behavior, an AUC of 0.94 is—at the very least—pretty solid.

As mentioned earlier, AUC is a threshold-independent measure. It evaluates the model’s ability to distinguish between classes by plotting sensitivity (recall) against the false positive rate (1 - specificity) across all possible thresholds. While this provides a useful summary of discrimination, it doesn’t reflect how the model performs at any specific cut-point.

In practice, classification models are often used to make discrete predictions—such as labeling a case as "bot" or "human"—and this requires selecting a threshold. Once a cut-point is chosen, we can assess the model's predictive performance using metrics like precision, recall, and predictive values. These metrics are dependent on that chosen threshold and offer a more practical view of how the model behaves in real-world decision-making.

Decision Table

The decision table below displays all predictive and ROC-based values across a range of cut-points.