The agent scanned the code and reported zero vulnerabilities — the code later turned out to have an obvious SQL injection. No error ever appeared. Why?
The silent failure of scope and config: the scan ran against a subdirectory or with an implicit minimal ruleset, matched nothing, and returned an empty findings list — which the agent interpreted as a security verdict instead of a coverage report. Semgrep never errors on low coverage; empty output is syntactically valid. The engineering fix: treat every clean result as suspicious until three numbers are confirmed — files scanned, rules run, and ruleset identity. A scan of 12 files with p/default proves nothing about a 4000-file repo. Pin the ruleset, scan the root, and have the agent report coverage statistics alongside findings, every single run.
Semgrep MCP vs asking the LLM to review the code directly?
Complementary, and the ordering matters. Semgrep finds with precision (low false positives, exact locations, no hallucination); the LLM triages with context (is this user input actually reachable?) and drafts fixes. LLM-only review invents findings and misses real ones; Semgrep-only dumps raw findings with no judgment. The pipeline is the product.
Can it write custom rules for our internal patterns?
Yes — Semgrep rules are YAML with pattern syntax, and the agent can draft them conversationally: flag any database call without our tenancy filter. Custom rules against internal conventions are where teams extract the most durable value.
Does it cover secrets detection?
The ruleset registry includes secrets rules (p/secrets), but dedicated scanners like Gitleaks or TruffleHog are stronger for credential-shaped entropy. Use Semgrep for code-pattern vulnerabilities and keep a dedicated secrets scanner in the pipeline.