Microbial communities in all environments are shaped by viral predators. Yet, resolving which viruses (phages) and bacteria are interacting, and how strongly viruses influence their potential hosts in the wild, remains a major challenge. We used large-scale isolation and cross-infection assays of environmental marine Vibrio bacteria and their phages to obtain quantitative estimates of strain-level phage predator loads and analyze the genetic underpinnings of interactions. We show that killing in environmental networks is typically highly specific and sparse with predator loads low on average. These observations are explained by receptors being host-species specific and host-strains being defended internally by diverse, large mobile genetic elements (MGEs) carrying phage defense genes. These MGEs are predominantly novel types and have high evolutionary turnover where a single strain can harbor as many as 12 different elements, overall accounting for a majority of the pangenome among closely related bacteria. Using interpretable machine learning and genetic assays, we also show that phages carry anti-defense genes that themselves show high evolutionary turnover and can be effective against multiple, unrelated defenses. Together, these results suggest a hierarchy in interactions where phage and host properties varying at different evolutionary timeframes result in sparse interactions at the strain level but highly modular interactions at the species level. Because these results question the assumption that viral predation is generally high, we are currently exploring when and where phage can exert effective control over bacterial populations.