Graded AI Evaluation Task & Environment Authoring (Harbor/Docker)
About this Gig
I design and build tamper-proof evaluation environments that measure whether an AI model or agent actually did the work — not just whether a string matched. • Harbor-format evaluation tasks with Dockerized, reproducible runtimes and launch scripts. • Oracle-vs-null verification: the task passes on a correct solution and fails on a broken one, proven both ways. • A mutant matrix where every wrong path is recorded actually failing, not assumed to. • Effect-based grading, hidden/held-out test separation, and agent-run trajectories. • Clear docs so anyone on your team can run and extend the task from a clean clone with one command. Public MIT reference: github.com/omair25-ozi/graded-env-reference. Backed by hands-on authoring of production evaluation tasks (Handshake AI / Project Dynamo). Python + Docker; fully written/async.
Requirements
The capability or behaviour you want to measure, your target domain (coding, agents, math, tool-use, etc.), any existing baseline or repo, your grading/reward format (e.g. Reward Kit or Harbor), and the difficulty level and number of task cases you need.
