The Agent That Got Slower the Longer It Ran: Two-Tier Memory for Long-Horizon Agents
The agent that remembered everything, and slowed to a crawl
Here is a failure mode that shows up on a schedule once you give an agent persistent memory. In week one it feels sharp: it recalls what you told it yesterday, it picks up a multi-day task where it left off, memory lookups are a few milliseconds and nobody thinks about them. By week six the same agent is noticeably slower on every turn, and — more insidiously — its answers have started drifting toward stale or irrelevant facts it dug up from months ago.
The reflex is to blame the vector index, or the model, or to move to a bigger database instance. Usually none of those is the real problem. The problem is architectural, and it is almost always the same one: all of the agent's memory lives in a single flat, append-only table that nothing ever consolidates or evicts. That design degrades on two axes at once, and you have to fix both.
Why a flat memory table degrades on two axes
The naive design is seductive because it is so simple. Every useful thing the agent observes — a user preference, a tool result, a conclusion — gets embedded and inserted into one table. Every turn, you embed the current context and pull the top-K nearest neighbours back as "memory." It works beautifully at small scale and then rots in two directions:
- Latency. Approximate nearest-neighbour search is sub-linear, not free. As the table grows from thousands to hundreds of thousands of rows, the HNSW graph gets deeper, the index no longer fits comfortably in cache, and recall-preserving search touches more nodes. The per-turn retrieval cost climbs, and because it runs on every turn, it sits directly in the user-facing latency budget.
- Relevance. This is the one people miss. A bigger memory pool means more old, near-duplicate, and contradictory entries competing for the top-K slots. Cosine similarity happily returns a six-month-old preference the user has since reversed, because it is semantically close to the query. The agent gets measurably worse at the same time it gets slower.
So "just raise k" or "just add a bigger index" treats a symptom. The fix is to stop treating all memory as one undifferentiated pool.
Two tiers: a hot working set and a consolidated long-term store
Human memory is the useful analogy here, and it is the shape most production agent stacks converge on. You split memory into a small, fast working tier that holds the current task's context, and a larger long-term tier that holds durable facts and is retrieved semantically. Critically, a background process moves information from the first into the second — and prunes as it goes.
The working tier: cheap, bounded, and ephemeral
Working memory is the last handful of turns for the current session or task. It is small by construction, so you do not need semantic search for it at all — a capped list in Valkey or Redis is enough, and it is sub-millisecond. The important part is that it is bounded and self-expiring, so it can never be the thing that grows without limit.
# Working memory: last 20 turns for a session, idle-expiring after an hour.
# LPUSH newest-first, LTRIM caps the list, EXPIRE refreshes the idle TTL.
LPUSH mem:{session_id} '{"role":"user","content":"..."}'
LTRIM mem:{session_id} 0 19
EXPIRE mem:{session_id} 3600
On each turn you read the whole list back with a single LRANGE mem:{session_id} 0 -1. No index, no ANN, no growth problem — the cap and the TTL guarantee it.
The long-term tier: semantic, but kept lean on purpose
Long-term memory is where durable facts live and where you actually need vector search. Model it as a plain table with an HNSW index, and — this is the part that pays off later — store the metadata you will need to rank memories, not just find them: an importance score, when it was created, and how often it has been used.
CREATE TABLE agent_memory (
id bigserial PRIMARY KEY,
agent_id text NOT NULL,
content text NOT NULL,
embedding vector(1536) NOT NULL,
importance real NOT NULL DEFAULT 0, -- 0..1, set at write time
created_at timestamptz NOT NULL DEFAULT now(),
last_accessed_at timestamptz,
access_count int NOT NULL DEFAULT 0
);
-- HNSW accelerates ORDER BY embedding <=> query only.
CREATE INDEX ON agent_memory USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON agent_memory (agent_id, created_at DESC);
Retrieval: let the index find candidates, then re-rank
The single most common mistake I see is pushing the full ranking logic into the ORDER BY:
-- SLOW: the ORDER BY expression is not a pure distance,
-- so the HNSW index can't be used. This becomes a full scan.
SELECT id, content
FROM agent_memory
WHERE agent_id = $2
ORDER BY 0.6 * (1 - (embedding <=> $1)) + 0.2 * importance + ...
LIMIT 12;
pgvector's HNSW index only accelerates an ORDER BY embedding <=> $query. The moment you order by a blended score, the planner falls back to scanning every row for that agent_id — exactly the cost that grows with the table. The fix is two-stage: use the index to pull a candidate set by pure vector distance, then re-rank that small set with your blend in an outer query.
-- Stage 1 (inner): HNSW-accelerated, returns ~100 nearest candidates.
-- Stage 2 (outer): cheap re-rank of those 100 on a blended score.
WITH candidates AS (
SELECT id, content, importance, created_at,
embedding <=> $1 AS distance
FROM agent_memory
WHERE agent_id = $2
ORDER BY embedding <=> $1 -- uses the HNSW index
LIMIT 100
)
SELECT id, content,
0.6 * (1 - distance)
+ 0.2 * importance
+ 0.2 * exp(-extract(epoch FROM (now() - created_at)) / 604800.0) AS score
FROM candidates
ORDER BY score DESC
LIMIT 12;
That recency term is a one-week (604800-second) exponential decay: a memory loses roughly 63% of its recency weight every week it goes untouched. The weights are a starting point, not gospel — tune them against your own eval set. The structural point is that re-ranking runs over 100 rows, not the whole table, so it stays flat as the store grows.
One more lever worth knowing: SET hnsw.ef_search = 100; trades recall for latency per query. Raising it recovers recall on a large index at the cost of touching more of the graph; it is the dial to reach for before you reach for a bigger instance.
Consolidation: the job that actually keeps it fast
Everything above still grows forever if nothing prunes. Consolidation is the background job that earns the "two-tier" name, and it does three things on a schedule (say, at the end of a session or hourly):
- Summarize and promote. Collapse a run of working-memory turns into one or a few durable facts, embed those, and write them to the long-term tier. You store the conclusion ("user is on the enterprise plan, prefers metric units"), not the raw transcript.
- Deduplicate. Before inserting, check whether a near-identical memory already exists (cosine similarity above, say, 0.95) and merge instead of inserting a third copy of the same fact. This is what stops the slow bloat of near-duplicates that poisons relevance.
- Decay and evict. Periodically drop long-term memories that are both low-importance and long-untouched. An append-only memory is a landfill; a store with a forgetting curve stays useful.
-- Evict cold, low-value memories. Run on a schedule.
DELETE FROM agent_memory
WHERE importance < 0.3
AND access_count <= 1
AND last_accessed_at < now() - interval '90 days';
Deduplication is the same two-stage trick as retrieval: for a candidate new memory, pull its nearest existing neighbour via the HNSW index, and if the distance is tiny, bump the existing row's importance and access_count rather than inserting. The dedup check costs one indexed lookup; skipping it costs you relevance for the lifetime of the agent.
What to measure
Instrument memory as its own span in every turn, separate from model time, so "the agent feels slow" becomes a number you can argue with. The three signals that matter:
- Memory retrieval p95, tracked against the long-term table's row count — if retrieval latency climbs with row count, consolidation is not keeping up.
- Long-term row count over time — it should plateau, not grow linearly. A straight line up means eviction is too timid.
- Retrieval relevance — on an eval set, how often the memories you pulled were actually used in the answer. This is the axis a flat table quietly destroys, and the one a bigger instance can never fix.
Lesson
Persistent memory is not a database you write to and forget; it is a system with a write path, a read path, and — the part that is easy to skip and expensive to skip — a forgetting path. A single flat vector table degrades on both latency and relevance at the same time, so adding hardware only buys you a slightly slower slide. Split the hot working set from the consolidated long-term store, let the index find candidates while cheap logic re-ranks them, and run a job whose entire purpose is to throw things away. An agent that forgets nothing is not smarter — it is just slower, and increasingly wrong.
Hitting something like this in production? I help teams with performance engineering, SRE/observability, and AI-driven root cause analysis — work with me.
Comments
Post a Comment