Articles / evalsfree-resourcesllmops
Free resources to learn LLM evals and LLMOps
Twenty free guides, courses, tools and papers for learning how to evaluate, trace and run LLM apps in production, in a sensible order.
Evals tell you whether a prompt or model change made your app better or worse. Online material on them is noisy: lots of tool marketing and benchmark tables, not much on how to test your app after a change.
I picked 20 resources from the GyaanStack evals & LLMOps and MLOps subjects (about 300 links together) and put them in the order I'd use them. I checked every link before publishing.
The short version of all of them: look at your data first, write tests for the failures you actually see, and only then reach for tools and judges.
Stage 1: why evals, and how to start#
Your AI Product Needs Evals. A widely shared post on why evals come first. It lays out three levels: unit-test style assertions, human and model review, and A/B tests. Start here.
AI Evals: Everything You Need to Know. A long FAQ covering the questions that come up once you start: how many examples, when to use a model as judge, how to handle subjective outputs. Read the sections you need.
LLM Evaluation: A Beginner's Guide (Evidently AI). A structured overview of evaluation methods, from reference-based metrics to LLM judges. Good for vocabulary.
“It's Hard to Eval” Is a Product Smell. Short. If you can't say what a good output looks like, the problem is usually the product spec, not the eval.
Stage 2: LLM-as-judge, used carefully#
Model-based grading is useful and easy to get wrong.
Using LLM-as-a-Judge For Evaluation: A Complete Guide. A step-by-step method: start with a domain expert's pass/fail calls and written reasons, then build a judge that agrees with them, and measure that agreement.
Evaluating the Effectiveness of LLM-Evaluators. A long survey of what the research says about LLM judges: which setups work, where they fail, and how to check them.
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (paper). The paper that made model judges mainstream, and also documented their biases: position, verbosity and self-preference.
Who Validates the Validators? (paper). Shows that people's grading criteria change as they look at outputs, so judge criteria need the same review loop. Useful background for any judge you build.
Consider pairwise evals instead of pointwise. Asking "which of these two is better?" is often more reliable than asking for a score out of 10. A short argument for when to switch.
Stage 3: RAG and agent evals#
Ragas: Automated Evaluation of Retrieval Augmented Generation (paper) and the Ragas docs. Metrics like faithfulness and context relevance for RAG. Read the paper for what the metrics mean, the docs to run them.
Demystifying evals for AI agents (Anthropic). How to grade agents: outcomes versus trajectories, code graders versus model graders, and building task sets.
AI Agents That Matter (paper). Argues that agent benchmarks ignore cost and overfit, and shows how to report accuracy together with cost. Read it before you trust any agent leaderboard.
Evaluating AI Agents (DeepLearning.AI short course). A hands-on course on tracing an agent, evaluating its components and adding experiments.
Stage 4: tools you'll actually use#
Pick one tracing tool and one eval runner. Don't install five.
Promptfoo. Config-driven evals you can run in CI, plus red-teaming for prompt injection and jailbreaks. A good first tool because it fits normal testing habits.
Inspect AI (UK AI Security Institute). An open-source eval framework the institute uses for its own model and agent evaluations, with a large set of ready-made evals in inspect_evals.
Langfuse docs. Open-source tracing, prompt management and evals. Self-hostable, which matters when your data can't leave your servers.
Arize Phoenix. Open-source tracing and evaluation that runs locally. Handy for debugging during development.
Stage 5: the ops side#
Evals are one part of running LLM apps. These cover the rest.
LLMOps (LLM Bootcamp) (Full Stack Deep Learning). A lecture on testing, monitoring and improving LLM apps after launch. Dated in its tool choices, still sound in its process.
Made With ML. A free MLOps course covering testing, CI/CD, serving and monitoring for ML systems. Not LLM-specific, but the engineering habits carry over directly.
Hidden Technical Debt in Machine Learning Systems (paper). The classic on why the model is a small part of a production ML system. Swap "model" for "prompt" and most of it still applies.
What I'd skip at first#
- Public benchmarks and leaderboards. They tell you how models compare on someone else's tasks. They don't tell you whether your app improved. Read AI Agents That Matter for why to be careful with them.
- Generic metric suites. A score called "helpfulness" from an off-the-shelf judge rarely matches what your users care about. Write criteria from your own failures instead.
- Buying a platform before you have a test set. Every tool above is free to start. The hard part is the 30 to 100 labelled examples, and no tool writes those for you.
- Grading on a 1–10 scale. Pass/fail with a written reason is easier to agree on, easier to check, and easier to turn into a regression test.
How I'd use this list#
- Week 1: Stage 1. Collect 30 real or realistic inputs for your app, read the outputs, and write down every failure.
- Week 2: turn those failures into code checks. Add a judge from Stage 2 only for what code can't check, and measure its agreement with you.
- Week 3: pick one tool from Stage 4 and run your checks in CI on every prompt change.
- Week 4: add tracing in production and feed new failures back into the test set.
For a working example without any API key, see my post on evaluating an AI agent in Python.
The full lists live in the GyaanStack evals & LLMOps (163 links) and MLOps (140 links) subjects. They're sorted into topics and updated daily.