DeepSeek R1

Executive summary. DeepSeek R1 shows critical risk in security and ethics; high risk in privacy, stereotype, toxicity, hallucination, and fairness; moderate risk in performance; and low risk in robustness.

Methodology. We measure vulnerability to attack along four dimensions — security, privacy, hallucination, robustness — and propensity for harm along four more: toxicity, stereotype, fairness, ethics. Each dimension is probed by modules of prompts designed to elicit a specific harmful behavior; a detector judges each response as pass or fail. Pass rates average into a dimension score, designated critical (0–50), high (51–70), moderate (71–90), or low risk (91–100).

Where it fails. R1 is highly prone to assisting in cyberattack, malware generation, XSS, and MITRE ATT&CK ontology attacks (several modules at a 0% pass rate). In privacy tests it reproduced masked proper names from copyrighted training text roughly one time in five. In hallucination tests it is weakest on misleading claims and social values, strongest on package hallucination.

Recommendations. If you customize this model: (1) fine-tune with additional data to improve robustness and reduce hallucination, (2) instruction-tune with selected prompt–response pairs to improve security, stereotype, and fairness scores, and (3) add guardrails for privacy, ethics, and toxicity.

Reproducibility. Results reproduce against DeepSeek R1 on any major model hub; we used the Together inference API. The full test suite is available by API, CLI, and web UI — start free and run it against your own deployment.

Vijil

Trust reports are produced by the Vijil evaluation team with Diamond, the trust-measurement engine. Reports are reproducible by any customer against the same model and test suite.