Articles / llmmemorygenai-basics
How chatbots remember a conversation: context windows, summaries and memory
The model is stateless; the app resends history each turn. Context windows, trimming, summaries, ChatGPT and Claude memory, and a Python demo with token counts.
Tell a chat assistant your name in the first message, ask for it ten messages later, and it answers correctly. It looks like memory. The model did not store anything, though. A language model is a function from input tokens to output tokens, and between two calls it keeps nothing. The chat app around it keeps the transcript and sends it back with every new message, so the model reads the whole conversation again each turn and predicts the next reply.
That design explains most of what users notice about chatbots. Long chats get slower and more expensive per message. Very long chats start to forget the beginning. Products then add "memory" features on top, which are another layer of text that the app chooses to put in the prompt.
This post walks through that machinery: what goes into each request, what the context window limits, how apps trim or summarize history, what ChatGPT and Claude say their memory features do, and then a small Python program that implements four strategies side by side, with real token counts for every turn and a recall test that shows where each one breaks.
The model is stateless; the app is not#
OpenAI's API guide on conversation state puts it directly: "each text generation request is independent and stateless." To hold a conversation, you include earlier turns in the next request. Anthropic's Messages API works the same way: you send a messages list with alternating user and assistant entries, and the context window docs describe each turn's input as "all previous conversation history plus the current user message."
So a chat with three turns produces three requests that keep growing:
turn 1 request: [system] [user 1]
turn 2 request: [system] [user 1] [assistant 1] [user 2]
turn 3 request: [system] [user 1] [assistant 1] [user 2] [assistant 2] [user 3]
\_________________ resent every time _________________/
When you ask "what is my name?" at turn 3, the answer comes from [user 1], which your app sent again. If your app had dropped it, the model would have no way to know.
Some APIs offer server-side state so you do not have to resend the list yourself. OpenAI has previous_response_id and a Conversations API. The same guide notes that with chaining, "all previous input tokens for responses in the chain are billed as input tokens." Server-side state saves you bandwidth and bookkeeping. You still pay for every earlier token on every turn, and the model still reads them.
What fills the context window#
The context window is the maximum number of tokens one request can hold. Everything counts toward it: the system prompt, tool definitions, every earlier message, retrieved documents, tool results, and the reply being generated. Anthropic's docs list a 1M-token window for current Claude models and 200k for some older ones (see context window sizes by model). If you are new to how text turns into tokens, What are tokens in LLMs? covers tokenizers and the cost math.
A typical support-bot request might be split like this:
system prompt and rules 1,200 tokens
tool definitions 800
retrieved help articles 2,500
conversation so far 6,000
current user message 150
reserved for the reply 1,000
------
11,650 tokens in one request
The conversation history is the only part that grows without limit. A pasted log file or a long code block adds hundreds or thousands of tokens in one message, so counting messages tells you little; count tokens. A chat with 100 short messages can be cheaper than one with 10 messages that contain stack traces.
A big window does not make history free. Anthropic's docs say that "as token count grows, accuracy and recall degrade, a phenomenon known as context rot," which is why curating what goes in matters as much as the window size. And on overflow, the Claude API does not quietly trim for you: if the input alone exceeds the window, it returns a 400 invalid_request_error ("prompt is too long"). Trimming is your app's job.
Three ways apps keep history under control#
Sliding window. Keep the system prompt and the most recent turns that fit a token budget; drop the oldest. It is cheap and predictable. Anthropic's docs note that chat interfaces such as claude.ai "can also manage the context window on a rolling 'first in, first out' basis." The cost is that early facts vanish once they scroll out.
Rolling summary. When history passes a threshold, replace the oldest turns with a short summary and keep recent turns word for word. Facts survive in compressed form. The summary can drop details, and writing it costs an extra model call. Claude's API now offers this server-side as compaction, where Claude writes the summary and you send it back in place of the older turns (both compaction modes are in beta). Claude Code's /compact command does the same thing for a coding session.
Retrieved memory. Pull durable facts (name, city, allergies, preferences) out of messages as they arrive, store them outside the conversation, and on each turn search the store for facts relevant to the current message. Only the hits go into the prompt. This scales to facts from weeks ago, but it depends on retrieval finding the right fact.
The agent memory system design post covers the production side of the third option: write policies, conflict handling, deletion and ranking. This post stays at the level of one chat app.
What ChatGPT and Claude say their memory does#
Product memory features are the third strategy, run by the vendor across chats. Here is what each vendor's help page states. Both pages say features vary by plan, region or workspace, so check them for your account.
ChatGPT. OpenAI's Memory FAQ describes two controls: saved memories and reference chat history. Saved memories "are stored separately from chat history," while information drawn from chat history "can change as ChatGPT updates what is most useful to remember." Three details from that page:
- Deleting a chat "does not necessarily delete a separate saved memory created from that chat." You delete the memory itself to stop it being used.
- Temporary chats "do not create or update memories, even with personalization on."
- Turning off reference chat history does not delete saved memories, and the information remembered from past chats "is scheduled for deletion from OpenAI systems within 30 days."
Claude. Anthropic's help article on chat search and memory describes two related features:
- Past-chat search is on paid plans (Pro, Max, Ultra, Team and Enterprise). The article says these searches "use Retrieval-Augmented Generation (RAG)" and show up as tool calls in the conversation, so you can see when Claude looks something up.
- Memory "saves memory as a set of individual topics as you chat." It is on by default for Free, Pro, Max and Ultra; on Team and Enterprise the organization owner controls it.
- Each project has its own memory, separate from other projects. Incognito chats are not saved to history and are not searched.
- Pausing keeps existing memory but stops using it and creating new entries. Resetting "permanently deletes all memories including project memories." You can view and edit topics under Settings, then Your Claude.
For developers, Anthropic's API has a memory tool that works the other way round: Claude asks to read or write files under /memories, and your application runs those operations against storage you control. Claude Code uses plain files for the same purpose: CLAUDE.md files load at the start of each session (see the Claude Code memory docs).
Every one of these features ends the same way. At request time, some selected text is added to the prompt. The model still starts from zero.
Build it yourself in Python#
The program below runs one scripted conversation through four strategies and prints the input tokens each strategy sends per turn. It uses OpenAI's o200k_base tokenizer from tiktoken 0.14.0 for counts (Claude uses a different tokenizer, so its counts would differ somewhat), plus a flat 4 tokens per message as a rough allowance for role markers.
There is no API key here. The model is a stub with the same shape as a real call: it receives the message list and answers only from text inside it, which is the property that matters for a memory test. The fact extractor that feeds the summary and the memory store is a regex stub too. In a real app both are model calls, and the comments show where they go.
Set up:
python3 -m venv .venv
.venv/bin/pip install tiktoken
The strategies#
memory.py:
"""Four ways a chat app can build the prompt for each turn.
The model never stores anything. Each strategy decides which text goes
into the next request.
"""
import re
import tiktoken
ENC = tiktoken.get_encoding("o200k_base")
PER_MESSAGE_OVERHEAD = 4 # rough cost of role markers per message
def count_tokens(messages):
return sum(len(ENC.encode(m["content"])) + PER_MESSAGE_OVERHEAD for m in messages)
# ---------- the model, stubbed ----------
class StubModel:
"""Answers only from text present in `messages`, like a real model would.
Swap `complete` for a real call, for example:
client.messages.create(model=..., system=system, messages=history, max_tokens=500)
"""
def complete(self, messages):
text = " ".join(m["content"] for m in messages)
last = messages[-1]["content"].lower()
name = re.search(r"my name is (\w+)", text, re.I)
city = re.search(r"I live in (\w+)", text, re.I)
allergy = re.search(r"allergic to (\w+)", text, re.I)
if "name" in last and "live" in last:
who = name.group(1) if name else "I don't know your name"
where = city.group(1) if city else "I don't know where you live"
return f"Name: {who}. City: {where}."
if "recipe" in last or "cook" in last:
if allergy:
return f"A {allergy.group(1).rstrip('s')}-free dal tadka with jeera rice."
return "Try a peanut satay stir-fry."
return "Noted. " + " ".join(messages[-1]["content"].split()[:10])
# ---------- a stub summarizer / fact extractor ----------
FACT = re.compile(r"[^.?!]*\b(my name is|I live in|I'm allergic|I am allergic|I prefer)\b[^.?!]*[.?!]?", re.I)
def extract_facts(text):
"""Stand-in for an LLM call such as:
'List durable facts about the user in this text, one per line.'"""
return [m.group(0).strip() for m in FACT.finditer(text)]
# ---------- strategies ----------
class FullHistory:
def __init__(self, system):
self.system, self.history = system, []
def build(self, user_msg):
self.history.append({"role": "user", "content": user_msg})
return [{"role": "system", "content": self.system}] + self.history
def record(self, reply):
self.history.append({"role": "assistant", "content": reply})
class SlidingWindow(FullHistory):
def __init__(self, system, budget):
super().__init__(system)
self.budget = budget
def build(self, user_msg):
msgs = super().build(user_msg)
# drop the oldest user/assistant pair until we fit; keep the system prompt
while count_tokens(msgs) > self.budget and len(self.history) > 1:
del self.history[:2]
msgs = [msgs[0]] + self.history
return msgs
class RollingSummary(FullHistory):
def __init__(self, system, budget, keep_last=4):
super().__init__(system)
self.budget, self.keep_last, self.summary = budget, keep_last, []
def _system(self):
s = self.system
if self.summary:
s += "\nSummary of earlier conversation:\n- " + "\n- ".join(self.summary)
return {"role": "system", "content": s}
def build(self, user_msg):
self.history.append({"role": "user", "content": user_msg})
while count_tokens([self._system()] + self.history) > self.budget and len(self.history) > self.keep_last:
old, self.history = self.history[:2], self.history[2:]
for m in old:
if m["role"] == "user":
self.summary += [f for f in extract_facts(m["content"]) if f not in self.summary]
return [self._system()] + self.history
STOP = set("a an the is are and or to of in on for my i i'm me do you can what where how so that this it be with".split())
def words(text):
return {w.rstrip("s") for w in re.findall(r"[a-z']+", text.lower()) if w not in STOP}
class RetrievedMemory(SlidingWindow):
"""Sliding window for recent turns plus a small fact store searched each turn."""
def __init__(self, system, budget, k=2):
super().__init__(system, budget)
self.store, self.k = [], k
def build(self, user_msg):
self.store += [f for f in extract_facts(user_msg) if f not in self.store]
q = words(user_msg)
scored = sorted(self.store, key=lambda f: len(q & words(f)), reverse=True)
hits = [f for f in scored[: self.k] if q & words(f)]
msgs = super().build(user_msg)
if hits:
msgs[0] = {"role": "system", "content": self.system + "\nKnown about the user:\n- " + "\n- ".join(hits)}
return msgs
Each strategy has the same two methods. build takes the new user message and returns the exact list that would go to the API. record stores the reply. The retrieval in RetrievedMemory is word overlap with a crude plural strip, kept small on purpose so the failure later is easy to see. A real store would use embeddings.
The conversation#
chat.py runs eight turns. Turns 1 and 2 hold the facts. Turns 3 and 5 paste a 276-token traceback and a 399-token requirements file, the kind of bulk that pushes early messages out. Turns 7 and 8 need the early facts. The budget for the trimmed strategies is 400 tokens, tiny on purpose so trimming happens within eight turns.
from memory import (StubModel, FullHistory, SlidingWindow, RollingSummary,
RetrievedMemory, count_tokens)
SYSTEM = "You are a helpful assistant. Answer briefly."
TRACE = "\n".join(f' File "app/routes.py", line {40 + i}, in handler_{i}\n result = service.call(payload)' for i in range(12))
REQS = "\n".join(f"package-{i}=={i}.{i}.0" for i in range(40))
TURNS = [
"Hi, my name is Priya and I live in Pune.",
"I'm allergic to peanuts, so keep that in mind for recipes.",
"My Flask app crashes. Here is the traceback:\n" + TRACE,
"Can you explain what a context window is?",
"Here is my requirements file, which ones are outdated?\n" + REQS,
"How does tokenization affect cost?",
"Suggest a recipe for dinner tonight.",
"What is my name and where do I live?",
]
BUDGET = 400
def run(strategy):
model, rows = StubModel(), []
for turn in TURNS:
msgs = strategy.build(turn)
reply = model.complete(msgs)
strategy.record(reply)
rows.append((count_tokens(msgs), reply))
return rows
if __name__ == "__main__":
strategies = {
"full": FullHistory(SYSTEM),
"window": SlidingWindow(SYSTEM, BUDGET),
"summary": RollingSummary(SYSTEM, BUDGET),
"memory": RetrievedMemory(SYSTEM, BUDGET),
}
results = {name: run(s) for name, s in strategies.items()}
print("input tokens sent per turn")
print("turn " + "".join(f"{n:>9}" for n in results))
for i in range(len(TURNS)):
print(f"{i + 1:>4} " + "".join(f"{results[n][i][0]:>9}" for n in results))
print("total" + "".join(f"{sum(r[0] for r in results[n]):>9}" for n in results))
for n in results:
print(f"\n[{n}]")
print(" turn 7:", results[n][6][1])
print(" turn 8:", results[n][7][1])
print("\nsummary kept:", strategies["summary"].summary)
print("memory store:", strategies["memory"].store)
Output from .venv/bin/python chat.py:
input tokens sent per turn
turn full window summary memory
1 30 30 30 49
2 60 60 60 79
3 365 365 365 365
4 401 371 390 371
5 831 427 489 427
6 869 24 498 24
7 894 49 82 68
8 923 76 111 97
total 4373 1402 2025 1480
[full]
turn 7: A peanut-free dal tadka with jeera rice.
turn 8: Name: Priya. City: Pune.
[window]
turn 7: Try a peanut satay stir-fry.
turn 8: Name: I don't know your name. City: I don't know where you live.
[summary]
turn 7: A peanut-free dal tadka with jeera rice.
turn 8: Name: Priya. City: Pune.
[memory]
turn 7: A peanut-free dal tadka with jeera rice.
turn 8: Name: Priya. City: Pune.
summary kept: ['Hi, my name is Priya and I live in Pune.', "I'm allergic to peanuts, so keep that in mind for recipes."]
memory store: ['Hi, my name is Priya and I live in Pune.', "I'm allergic to peanuts, so keep that in mind for recipes."]
Reading the numbers#
Full history is correct and sends 4,373 input tokens over eight turns. Every turn after turn 5 carries both pastes again, even though nobody asks about them. At a long chat's scale (hundreds of turns, real code), that resending is where the bill goes.
The sliding window sends 1,402 tokens, about a third, and fails both memory questions. By turn 6 it had dropped everything older than the current message, so it suggests peanut satay to someone allergic to peanuts. Plain trimming fails this way in production: the answer is fluent, confident and wrong, and nothing in the reply signals that context was lost.
The rolling summary sends 2,025 tokens and answers both questions. Its turns 5 and 6 (489 and 498) go over the 400 budget, because keep_last=4 refuses to fold the four most recent messages and two of them are the big pastes. A budget is only a budget if something enforces it on single large messages too.
The retrieved memory sends 1,480 tokens and answers both. It costs a little more than the window on turns 1 and 2 (49 against 30) because it injects the fact it just extracted, a duplicate of the message already in the window. Then it pays for itself: at turns 7 and 8 it adds one relevant fact instead of the whole history.
Turn 5 also shows a case the code does not handle. The window strategy sent 427 tokens, over budget, because the requirements file alone nearly fills 400 and the loop stops when one message is left. A production app has to cut or summarize an oversized message itself; against the real API, a message bigger than the whole window gets a 400 error.
Test that memory still works#
Token savings mean nothing if facts go missing, so the last file is a recall test. It plays the first six turns, then asks three probe questions from the same state, one of them phrased without any word the stored fact contains.
test_recall.py:
"""Does each strategy still carry the facts from turns 1-2 after the long pastes?"""
import copy
from chat import TURNS, SYSTEM, BUDGET
from memory import StubModel, FullHistory, SlidingWindow, RollingSummary, RetrievedMemory
PROBES = {
"Suggest a recipe for dinner tonight.": ["peanut-free"],
"What should I cook tonight?": ["peanut-free"],
"What is my name and where do I live?": ["Priya", "Pune"],
}
def warm(strategy):
model = StubModel()
for turn in TURNS[:6]: # everything before the questions
strategy.record(model.complete(strategy.build(turn)))
return strategy
for name, s in {"full": FullHistory(SYSTEM), "window": SlidingWindow(SYSTEM, BUDGET),
"summary": RollingSummary(SYSTEM, BUDGET), "memory": RetrievedMemory(SYSTEM, BUDGET)}.items():
base, passed = warm(s), 0
for probe, expected in PROBES.items():
trial = copy.deepcopy(base) # each probe starts from the same state
reply = StubModel().complete(trial.build(probe))
ok = all(e in reply for e in expected)
passed += ok
if not ok:
print(f" FAIL {name:<8} {probe!r} -> {reply!r}")
print(f"{name:<8} {passed}/{len(PROBES)}")
Output:
full 3/3
FAIL window 'Suggest a recipe for dinner tonight.' -> 'Try a peanut satay stir-fry.'
FAIL window 'What should I cook tonight?' -> 'Try a peanut satay stir-fry.'
FAIL window 'What is my name and where do I live?' -> "Name: I don't know your name. City: I don't know where you live."
window 0/3
summary 3/3
FAIL memory 'What should I cook tonight?' -> 'Try a peanut satay stir-fry.'
memory 2/3
The memory store holds the allergy fact, yet "What should I cook tonight?" shares no content word with "I'm allergic to peanuts, so keep that in mind for recipes," so retrieval returns nothing. Embeddings narrow this gap but do not close it. Two fixes are common: put a small, always-on user profile (name, city, allergies) in every prompt instead of retrieving it, and run probes like these in CI whenever you change the extractor, the retriever or the budget. The agent memory post goes into the profile-versus-retrieval split in detail.
To run the same test against a real model, replace StubModel.complete with an API call and swap the exact-string checks for a looser check or an LLM judge, since a real reply will not say "peanut-free" word for word every time.
Failure modes to watch for#
- Silent forgetting. Trimmed history produces confident wrong answers, as the window run shows. Log when trimming drops a turn.
- One giant message. A single paste can exceed the budget on its own. Cut it, summarize it, or store it and pass a reference.
- Summary drift. Each re-summary is lossy, and a summary of a summary loses more. Keep exact values (IDs, amounts, dates) in structured fields, not prose.
- Retrieval misses on paraphrase. Shown above. Test with phrasings that do not reuse the user's words.
- Stale facts. "I live in Pune" from March loses to "we moved to Hyderabad" in October only if your store knows how to replace a fact. Appending both confuses the model.
- Cache misses. Editing the start of the prompt, by trimming or by rewriting a summary, breaks prompt caching. The token-saving guide for Claude covers how to batch summarization so the cache survives.
- Users who expect deletion. Deleting a chat may not delete what was learned from it, as OpenAI's FAQ states for ChatGPT. If you build memory, give users a way to see and delete it.
Sources#
- OpenAI, Conversation state (stateless requests, chaining and billing)
- OpenAI Help Center, Memory FAQ
- Anthropic, Context windows
- Anthropic, Compaction overview
- Anthropic, Memory tool
- Anthropic Help Center, Use Claude's chat search and memory to build on previous context
- Claude Code docs, How Claude remembers your project
- tiktoken on GitHub