GyaanStack. Blog

Articles / agentslanggraphmcpprojects

CareerPilot: what I learned building a five-agent job-hunt copilot with LangGraph and MCP

How I built a multi-agent job-hunt copilot with LangGraph, MCP, RAG and read-only SQL, plus the bugs my eval harness caught.

By Raghunath · · 11 min read

Job hunting comes down to a few repeated questions. Does my resume fit this job description? Which companies are hiring for this role right now? How many of my applications got a reply, and which ones went quiet? Can you draft a follow-up to that recruiter?

Each question needs a different kind of system. The first is retrieval over my own documents. The second needs live data from job boards. The third is SQL over a tracker. The fourth writes text that will leave the app, so it needs a human to approve it.

CareerPilot puts all four behind one chat box. You type a question in plain English, a router picks the specialist agents that should handle it, they run in parallel, and you get one combined answer. Every routing decision and tool call streams into a trace panel so you can see what happened.

The live demo is at careerpilot-raghu298.netlify.app and the code is at github.com/raghu298/careerpilot. This post walks through how it works and the bugs I hit along the way.

The five agents#

Agent What it answers How
Knowledge "Does my resume fit this JD?", "What ATS keywords am I missing?" RAG over your resume, the target JD and career guides
Jobs "Find GenAI roles", "What do senior GenAI roles pay?" Tool calling over a jobs MCP server backed by live RemoteOK and Remotive postings
Analytics "How many applications got a response? Which are stale?" Natural language to read-only SQL over your application tracker
Notify "Draft a follow-up to the recruiter at Acme" Drafts the message and stages it; nothing goes out until you confirm
Interview "Prep me for the interview" A mock-interview plan grounded in your resume and the JD

One request can need several of them. "Find a role I haven't applied to yet and draft a cover letter for it" touches Jobs, Analytics and Notify at once.

Routing: let the model reason, but force the answer into a schema#

The router is an LLM call. I don't trust free text from it, so its output has to validate against a Pydantic model before anything downstream sees it. This is the whole schema, from src/careerpilot/orchestrator/schemas.py:

AgentName = Literal["knowledge", "jobs", "analytics", "notify", "interview"]


class RoutingDecision(BaseModel):
    agents: list[AgentName] = Field(min_length=1, ...)
    reasoning: str = Field(
        description="Short justification, kept for tracing/eval. Not shown to the user."
    )

If the model returns an agent name that doesn't exist, or an empty list, validation fails and the router retries, up to two more times. After that it raises a RoutingError instead of guessing.

The schema stops invalid answers. It doesn't stop a valid answer that names the wrong agent, and that is the system prompt's job. "Salary" is the clearest example: "what do GenAI roles pay" needs live market data (Jobs), but "how should I negotiate my salary" needs the career guides (Knowledge). The prompt says that in one line: "Salary NEGOTIATION advice is knowledge; market salary DATA is jobs." Each agent's description also says what it does not do. The Jobs agent "does NOT know the user's resume or which jobs they've already applied to." I keep the descriptions mutually exclusive so two agents never compete for the same keyword.

The graph: fan out, merge, pause#

The orchestrator is a LangGraph StateGraph with three nodes. From src/careerpilot/orchestrator/orchestrator.py:

def _build_graph(self) -> StateGraph:
    g = StateGraph(GraphState)
    g.add_node("router", self._router_node)
    g.add_node("run_agent", self._run_agent_node)
    g.add_node("aggregate", self._aggregate_node)
    g.add_edge(START, "router")
    g.add_conditional_edges("router", self._fan_out, ["run_agent"])
    g.add_edge("run_agent", "aggregate")
    g.add_edge("aggregate", END)
    return g

def _fan_out(self, state: GraphState) -> list[Send]:
    return [
        Send("run_agent", {"agent_name": name, "user_request": state["user_request"],
                           "history": state["history"], "results": {}})
        for name in state.get("selected", [])
    ]

_fan_out returns one Send per selected agent, so LangGraph runs those run_agent invocations concurrently. Each one writes {name: result} into results, and a dict-merge reducer on that field combines them. The graph only knows about an Agent protocol and a name-to-agent dict, so adding the fifth agent (Interview) meant writing the agent, adding its name to the Literal and describing it in the router prompt. orchestrator.py didn't change.

A reducer bug I didn't see coming#

The results channel is checkpointed per session thread with a MemorySaver, and the reducer merges into whatever is already there. So on the third turn of a conversation, results still held the replies from turns one and two, and the app showed them again. The fix was small. _finalize now returns only the agents selected for the current turn, and I added a regression test that runs three turns and checks that each one stays isolated. If you put a merge reducer on a checkpointed channel, remember that it keeps merging across turns.

Human-in-the-loop for the one agent with side effects#

Notify is the only agent that can cause something to happen outside the app, so it never sends anything from handle(). It drafts the message with the LLM (schema-guarded again), stages it as a PendingAction, and returns. The recipient comes from configuration. The model never picks it.

The aggregate node checks whether any agent has something staged and, if so, pauses the whole graph:

if isinstance(agent, ConfirmableAgent) and agent.has_pending():
    pending = agent.pending()
    emit(on_event, {"type": "pending_action", "agent": name, "kind": pending.kind})
    # Pause the graph for a human decision (Send / Edit / Cancel).
    decision = interrupt({"agent": name, "pending": _pending_to_dict(pending)})
    resolved = await self._apply_decision(agent, decision)

LangGraph's interrupt() saves the graph state to the checkpointer and returns control to the API. The UI shows a card with Send, Edit and Cancel. Whichever you pick, the API calls resume_pending, which runs ainvoke(Command(resume=decision)) on the same thread, and the graph continues from where it stopped. Edit sends your instructions back to the LLM to revise the draft. The agent also scans the draft for personal data (emails, phone numbers, card numbers) and adds a warning to the preview if it finds any.

In the demo, "Send" writes to a Demo Outbox page instead of a real inbox. Setting a Slack or Resend key switches it to a real adapter.

The Jobs agent talks to a real MCP server#

I wanted the Jobs agent to use the Model Context Protocol for real, not a function that only looks like MCP. mcp_server/jobs_mcp_server.py is a standalone MCP server with four tools: search_jobs, get_company, salary_insights and trending_skills. It pulls live postings from RemoteOK and Remotive, which need no API key. The backend starts it as a stdio subprocess and calls it through the official mcp SDK.

Salary figures come from postings that actually list pay. salary_insights takes the median of the low and high ends across matching postings and drops amounts outside a sane annual range, because RemoteOK mixes in hourly and placeholder figures that dragged the band to "$0k". If nothing matches, it says so instead of producing a number.

The client opens a fresh MCP session per tool call. A long-lived session triggered anyio cancel-scope errors when the call started inside a LangGraph node, and a session per call keeps each interaction inside the task that made it.

Production quietly served fake jobs#

There is also a FakeJobsMCP with four hardcoded companies so tests run offline. At startup the app probes the live server and falls back to the fake if the probe fails, so the app always boots.

On Render's free tier, a cold start meant booting the subprocess plus a slow first fetch from RemoteOK, which took longer than the 15-second probe timeout. Production fell back to the fake and stayed there. Users would have seen invented companies with no error anywhere. I raised the timeout to 45 seconds and added one retry. A fallback you can't see needs to be logged and monitored, because otherwise nothing tells you it's running.

Read-only SQL with three layers#

The Analytics agent has one tool, run_sql_query, and the live schema goes into its prompt so the model doesn't guess column names. Its SQL has to get past three guards:

  1. The prompt tells it to only write SELECT statements. Nothing enforces that.
  2. A regex filter rejects anything that isn't a single read-only statement before it reaches the database.
  3. In production, a read-only Postgres role. This is the only layer that application code can't bypass.

The filter is short. From src/careerpilot/integrations/sql_db/postgres_client.py:

def validate_readonly(query: str) -> str:
    stripped = query.strip().rstrip(";").strip()
    if ";" in stripped:
        raise ReadOnlyViolation("Multiple statements are not allowed")
    if not stripped.lower().startswith(("select", "with")):
        raise ReadOnlyViolation("Only SELECT queries are allowed")
    if _WRITE_KEYWORDS.search(stripped):
        raise ReadOnlyViolation("Write/DDL keywords are not allowed")
    return stripped

The regex is a fast first check, not the security boundary. I keep it because it fails early with a clear message the agent can read and recover from.

Access control works the same way: the code decides it, not the prompt. There are two demo personas. Arjun can see premium salary-negotiation and leadership-resume guides. Riya can't. That rule lives in the vector search itself, as WHERE access_scope = ANY($2) in the pgvector query, so a prompt injection can't talk the model into retrieving a document the persona isn't allowed to see.

Evals caught three bugs on the first run#

The evaluation/ folder has two harnesses that run offline with no API key:

  • run_knowledge_eval checks each golden question for whether the expected source gets cited and whether answer_found matches what it should be. It also computes Recall@1, @3 and @5 plus MRR from the retrieval ranking alone, with no LLM judge. If OPENAI_API_KEY is set and DeepEval is installed, it adds faithfulness and contextual precision/recall metrics.
  • run_agent_eval checks Jobs results for topical relevance (not exact titles, since live data changes), checks Analytics answers for the right numbers, and confirms that DELETE, UPDATE and DROP are rejected.

The first run failed. An off-topic question about a rocket to Mars got answered from the resume documents. Riya, who has no salary guide in scope, got answer_found=True from unrelated chunks. And the Analytics agent only ever ran a status rollup, so "which applications are pending" ignored the filter. I fixed the first two with a grounding check: the agent only answers from a chunk that shares a content word with the question, and otherwise reports not-found. The third needed real WHERE status IN (...) filters.

None of the unit tests had caught these, because each test checked one agent doing what I expected. The golden set asked questions I hadn't expected.

Ports and adapters made the offline mode possible#

Agent code never imports a vendor SDK. LLMClient, EmbeddingClient, VectorStore, SQLClient and JobsSource are protocols, each with a real adapter and a fake. Without an OpenAI key, the app uses a deterministic offline stand-in: a small rules engine that satisfies the same protocols. It isn't a model, but it means a fresh deploy works with no keys, and the test suite (more than 60 tests) runs with no network.

The frontend is a static Next.js export on Netlify. The FastAPI backend runs on Render's free tier because it holds long-lived SSE connections, which Netlify's serverless functions don't suit. The backend sleeps after about 15 minutes idle, so the page warms it on load and retries through the 30 to 60 second cold start.

One SSE bug was hard to spot. The server separated events with \r\n\r\n and the browser client split on \n\n. curl showed perfect output while the browser rendered nothing. The client now converts CRLF to LF before splitting.

What I'd do differently#

I'd write the golden eval set on day one, before the second agent. Its first run found three real bugs that the unit tests had missed, and the cases were cheap to write.

I'd also make every fallback loud from the start. The fake jobs source and the offline brain are both useful, and both can hide a broken production setup if nothing reports that they are active.

To try it, open the demo, pick Riya or Arjun (no sign-up), and ask something that needs two agents, like "find GenAI roles and draft a cover letter for the first one." Watch the trace panel split into parallel branches, then confirm or cancel the draft. To run it locally, clone the repo, pip install -e ".[dev]", and run pytest -q and python -m evaluation.run_knowledge_eval. Neither needs a key.