From the source
This research explores whether AI-driven security scans are sufficient for vibe coding platforms , or whether they risk asking models to audit their own output.
Through controlled experiments on React applications with realistic vulnerability variants, we compare AI-only security scans with Replit’s hybrid approaches that combine deterministic static analysis and dependency scanning with LLM-based reasoning.
Along the way, we examine how prompt sensitivity, nondeterminism, and ecosystem awareness affect real-world security outcomes.
We show that functionally equivalent code can receive different security assessments depending on syntactic form or prompt phrasing.
Issues like hardcoded secrets may be detected in one representation and missed in another.
More critically, dependency-level vulnerabilities and supply-chain risks remain largely invisible without traditional scanning infrastructure.
The takeaway is not that LLMs are ineffective, but that they are best used alongside deterministic tools .
…
