The new Jev-as-a-judge functionality enables AI developers to analyze full production datasets for safety and security flaws instead of relying on small samples. This integration into the LangSmith platform allows for more criteria per trace without increasing costs and can trigger automated responses to critical security issues. The tool utilizes a specialized class of decision models that provide typed answers, such as probabilities and rubric scores, rather than generative text.

LangSmith Gateway is also offering free access to SemIf, an open source decision model based on Qwen 3.5, for 1 week starting Sept 21. SemIf scored 75.4 on the JevBench benchmark compared to 74.7 for Jev and remains compatible with the TypeSafe SDK. Other open source alternatives to Jev are also available to developers for trace scoring and evaluation.