Organic photodetectors (OPDs) exhibit superior spectral responses but slower photoresponse times compared to inorganic counterparts. Herein, we study the light-intensity-dependent OPD photoresponse time with two small-molecule donors (planar MPTA or twisted NP-SA) co-evaporated with C60 acceptors. MPTA:C60 exhibits the fastest response time at high-light intensities (>0.5 mW/cm2), attributed to its planar structure favoring strong intermolecular interactions. However, this blend exhibits the slowest response at low-light intensities, which is correlated with biphasic photocurrent transients indicative of the presence of a low density of deep trap states. Optical, structural and energetical analyses indicate that MPTA molecular packing is more strongly disrupted by C60 than NP-SA, resulting in a larger (370 meV) HOMO level shift. This results in greater energetic inhomogeneity, leading to deep trap state formation which limits the low-light photoresponse time. This work provides important insights into the organic small molecule design rules critical for low charge-trapping and high-speed OPD applications.