ML / AI · Jun 2026
VLM Evaluation Harness, Multimodal Model CLI
A command-line evaluation harness for vision-language models that emits structured JSON and CSV logs per run. The tool is designed for reproducible multimodal benchmarking with explicit configuration, deterministic seeds, and per-sample traceability. Metrics now include exact-match and SQuAD-style token-F1 with per-example F1 traceability and a constant-baseline reference.
CLI
Surface
JSON, CSV
Outputs
VLMs / MLLMs
Target
Problem
Multimodal model evaluation is fragmented across ad hoc notebooks, with results that are difficult to compare or reproduce. A lightweight, scriptable harness was required to standardize runs and log every prediction alongside its configuration.
Approach
I built a config-driven CLI that loads a model and dataset, runs evaluation with deterministic seeds, and writes per-sample JSON alongside aggregate CSV. The design favors composability over a monolithic framework, allowing new tasks and metrics to be added without modifying the core loop.
Results
The harness produces a self-contained evidence trail for every evaluation run, enabling straightforward diffing across models and configurations and serving as the substrate for downstream VLM studies.
Stack