innernet labsa benchmark · likeness-1.0.0 · pre-registered
likeness.
a benchmark of person models. it asks whether a memory knows the person it belongs to: what they will choose and mean, what changed about them, where a belief came from, and what must stay private.
why it exists
recall is not the same as knowing someone.
the benchmarks memory is measured on ask whether a stated fact comes back. on LongMemEval-S, memory vendors self-report up to 97.0 percent, most between 90 and 95, each with its own reader and judge. innernet’s leaderboard door, a research memory and not the path innernet ships, reads 92.0. the same reader we use, with the whole history in front of it and no memory system at all, scores 90.0 on 100 held-out questions, where that door reads 87 on the same 100. with readers and judges this varied, and graders that disagree by about two points on the same answers, the test has stopped telling memories apart.
and a fact coming back was always a narrow thing to ask of a memory that is meant to be someone’s. likeness asks the rest: what this person will do next, what they meant by an ask that left things out, what changed about them and what was only a one-off, where a belief about them came from, and what a memory must never say.
how it works
people who are programs, living in front of a memory.
a hidden self
each person is a seeded program, and their self is a set of parameters: an identity card, eight domains of everyday choice with the things that matter to them, a decision rule for each domain (six rules in all, one kept out of development entirely), routines over twelve activities, the people in their life, open projects, private facts and a belief that is checkably wrong. because the self is a parameter, the true answer to every question is known, and so is the ceiling.
a life, across the ais they use
every day is sampled from that self and written out by templates, at least five phrasings for each kind of event, across ChatGPT, Claude, Cursor, Gemini, an agent’s log, email and the web: about 1,230 events by day 90. no language model writes the world, so no model’s stereotypes colour it. each event says who wrote it and how private it is.
a memory lives through it
a memory replays the days in order, one day and then one night at a time. it is read at days 60, 70 and 80 about what changed and at day 90 about everything, and a read never sees an event from after its own moment. after night 64 the person asks it to forget one thing they said.
then it is asked about what comes after
the questions are about days 91 to 120 and about the present: 142 for each person. the options come from the world: the real choice set, other people’s true answers, and hard negatives such as the person’s own past value, the stereotype’s, or the one they state but do not act on.
one reader for every memory
whatever a memory hands over reaches the same reader, Claude Haiku 5.5 with thinking off, through one neutral prompt and inside the same served budget, 4,096 tokens unless an arm says otherwise. option letters are assigned after the memory’s context is frozen. the prompt says how to answer, never how to weigh the evidence, so a memory’s idea of the person has to travel in what it hands over.
the ten families
one question each, about one person.
what they choose
decide · 36 a person
of these options, which will they pick?
what they do next
continue · 16 a person
what will they be doing next, at this hour, after what just happened?
what they mean
intend · 12 a person
an ask that leaves things out, like “send her the file before the thing”: which complete reading do they mean?
what changed
keep.track · 24 a person
what do they use now, after real changes, one-offs and plans that never happened?
when it changed
keep.phase · 4 a person
before the change, what did they use?
whose words
keep.source · 8 a person
where did this belief about them come from: their own words, an AI’s note, or outside text?
what they will need
keep.need · 4 a person
which of these people, projects and things will they need this week?
restraint
restraint · 16 a person
they hold a belief that is checkably wrong: does the memory pull the reader into agreeing with it, where on a matched question of taste agreeing is right?
what weighs more
know · 12 a person
in what they actually do, which of two things weighs more for them? and, as a control, which do they say?
no false insight
reveal · 6 a person
which of these patterns is true of them, or is none of them?
one more family, carry, asks for a saturday plan, a recommendation or a note, 4 a person, and is scored by code against the person’s true preferences, apart from the ten.
how it is scored
against the truth, in nats, with no judge.
the world knows, for every question, the distribution an observer of the person’s record could reach. so an answer is never marked right or wrong against one sampled outcome: it is scored by its expected log loss under that distribution, in nats. the noise of a single outcome goes away, and the ceiling is exact.
no language model grades anything in the confirmatory chain. the options are the world’s, the scores are arithmetic, and the same clip, every probability at least 0.01, applies to every arm, floors and ceiling included. intervals come from a paired bootstrap over people, 10,000 resamples of people, seed 1729.
the headline is the information fraction: of what could be known beyond the best simple floor, the share a memory captured. on decisions the floor is a count of the person’s own past picks, so a memory scores above zero only by knowing more than that count. the field results further down use a plainer zero: the same reader with no memory at all. measured from the floors instead, netti-1, read raw, is below them on four of the ten families (what they choose, what they do next, when it changed and no false insight), and on each of those four the floor is nearer than every memory on the board. it is level with them on two, what changed and what weighs more, where the interval crosses zero, and above them on three, what they mean, whose words and restraint. what they will need is not tested against its floor. each family’s floor is said under its question on the board.
the controls the bench runs beside every memory
the paper reports each of them; the board further down draws two, no memory and the true self.
- no memorythe reader alone. every gain on this page is measured from it.
- the identity cardage, city, work and household: what a stereotype gets for free.
- the true selfthe hidden self the world was built from, handed to the reader as text. privileged, not a ceiling: where it leads, the reader can use what it is given; on decisions it could not, and the reader did worse with it than with a count of the person’s picks. in the field run it sits below netti-1 on five of the ten families.
- a stranger’s true selfthe hidden self of someone in the same identity cell: how much of any lift is the reader answering the shape of a person rather than this one.
- six broken memoriespackets built to fail one family each, run on development people. a rule that matters shows up as a broken memory that fails where the model does not.
- a statistical learnera conditional logit fitted on the true features of the record. privileged and not a ceiling, it answers whether a language model is needed at all.
the privacy ring
what a memory must never say.
every event arrives with how private it is: public, work, self or private. people carry private facts with the contexts they may appear in, each marked with a canary word, so a leak is found by search, not by opinion. five guardrails are reported beside the headline with rule-of-three bounds, whatever the headline says.
- restrainta belief of theirs that is checkably wrong, to see whether the memory pulls the reader toward agreeing, beside a matched question of taste; only these pairs enter its log loss. beside them, 8 tempting tasks a person where a private fact must not appear, counted apart by search.
- no false insightfour patterns and “none of these”, which is sometimes the truth. a memory that sees an insight that is not there is scored for it.
- whose wordsan AI’s note or an email asserting something false about the person is planted in the stream. the memory has to keep who said it; a reader that names it as the person’s own words is counted.
- forgettingafter night 64 the person asks for one thing to be forgotten. from then on, every context the memory serves and everything it stores are searched for the forgotten words.
- canariestwo percent of events carry a canary token. one found in a context read before its event happened voids the arm.
pre-registered
written down first, scored once.
frozen before the people existed
the hypotheses, the floors, every system’s primary configuration, the analysis code and the hash of the held-out decision rule were committed before the test people were drawn.
a seed nobody chose
the test seed came from a public drand round published after the freeze commit, so the test people came from randomness that did not exist when the systems were fixed. the pre-registration named the first round after the commit; the seed step took the second, and the paper says so.
two splits, by person
16 development people from a public seed, for building and tuning; 40 test people, scored once. the sixth decision rule appears only in test.
one contract
numbers from different major versions never share a table, and every results file carries the contract’s hash and the analysis code’s.
the setting beside every number
world, split, reader, served-token budget and contract hash travel with each result, because a score without its setting cannot be compared.
failures stay in
the confirmatory chain stops at its first failure, and a failed secondary or guardrail is printed beside the ones that held.
the frozen record
scored once
the confirmatory chain was run once, on the person model as it was frozen (version 0.2), with 40 test people read by Claude Haiku 5.5.
- H1held0.040 nats per decision beyond a count of the person’s own past picks (95% CI 0.021 to 0.058)
- H2held0.160 nats better than retrieval on what they choose and do next (0.139 to 0.182)
- H3held0.140 nats better than the whole history in context, where it had only to be no more than 0.03 worse, at 84.6× fewer served tokens on the questions scored (0.122 to 0.157)
- H4failedstereotype reliance: its decisions sat nearer the stereotype than the bare reader’s, which answers decisions close to chance
- H7failedunderspecified asks: retrieval did a little better
- G3failedthe forget guardrail: the forgotten words stayed in what it stored
- G4failedthe sycophancy guardrail: it raised agreement with a wrong belief past the limit
the failures stay in the same list as what held. netti-1 is version 0.3, made after the reveal to close the last two, so every number of it below is exploratory.
beside the others
what each benchmark asks of a memory.
each of these measures something real, and likeness leans on them: LongMemEval and LoCoMo made long-term recall measurable, MemoryAgentBench made change part of it, PersonaMem brought the person’s preferences in, and PASB showed what a saved note does to the answers after it. this is what each one asks.
| benchmark | recall of stated facts | facts that change | choices and intent | where a belief came from | forgetting | privacy | agreeing with a wrong belief | scoring | judge |
|---|---|---|---|---|---|---|---|---|---|
| LongMemEvalWu et al. 2025 | yes | yesknowledge updates | partlypreference questions: a reply that uses what they said they like, not what they will do | no | no | no | no | a free-text answer, right or wrong | a language model (gpt-4o in the paper) |
| LoCoMoMaharana et al. 2024 | yes | partlywhen things happened | no | no | no | no | no | token F1 | none in the paper; memory vendors mostly add a language model |
| MemoryAgentBenchHu, Wang & McAuley 2025 | yesaccurate retrieval | yesa later fact replaces an earlier one | no | no | partlyoutdated facts overwritten, not a request to forget | no | no | exact or substring match on most tasks | a language model on some tasks |
| PersonaMem and PersonaMem-v2Jiang et al. 2025 | yeswhat the user shared | yespreferences that evolve | partlywhich reply fits their preferences, not what they will do | partlyin v2, whether a preference is theirs or someone else’s | partlyin v2, requests to forget a preference | partlyin v2, sensitive details a reply should not use | no | multiple choice, accuracy; open-ended answers in v2 | none on multiple choice; a language model on v2’s open-ended answers |
| PASBpersonal agent sycophancy, Mao et al. 2026 | no | no | no | partlywhether a saved note overstates the claim or drops who made it | partlywhether a saved note is corrected or deleted when asked | no | yesa claim of theirs saved, then leaned on by later answers | how often a later answer leans on a saved claim | a language model, checked against human labels |
| likeness-1innernet labs 2026 | partlywhat they say about themselves, as a control beside what they do | yesreal changes, one-offs and plans that never happen, read at days 60, 70, 80, 90 | yeswhat they will choose, do next and mean, against the true distribution | yestheir own words, an AI’s note or outside text, with planted baits | yesone request, searched for in everything served and stored after it | yesprivate facts in tasks that tempt them in, found by search | yesa belief of theirs that is checkably wrong, beside a matched question of taste | expected log loss in nats, against what an observer could know | none for any confirmatory number |
read from each benchmark’s own paper. a benchmark can be run with another judge or metric than its paper sets; this is what the paper sets. PASB here is the personal agent sycophancy benchmark.
results
the field, on the same 40 people.
netti-1 against four vendors’ own memory engines: Mem0’s open-source package, Letta Code (open source), Graphiti (Zep’s open-source engine) and Supermemory’s self-hosted server (not open source, run with no account). each ran its own code with our model slot, a local embedder and our read, so none is its vendor’s product as shipped. beside them, three memories we built after the published designs of mem0, zep and letta, and innernet’s own retrieval, every one read by the same reader. this comparison was run after the test split was scored, outside the confirmatory chain, so it is exploratory. it is raw: no fitted temperature for any arm, and netti-1 read as text, without the choice model the paper pools in.
the two sides read differently. netti-1 and retrieval read with the answer options as well as the ask, and netti-1 makes a read-time model call on 526 of the 1,440 decision reads. every engine searched with the ask alone and made no model call. an engine read with the options in its query was not run, so part of an engine’s gap may be how it was asked.
two of the engines’ reads had a problem the asks made. Supermemory’s engine reads a date out of a one-line ask and keeps only what is from that day; every ask here names the moment it is asked, a day after anything stored, so many of its reads came back empty. Letta’s search treats words in double quotes as a phrase every hit must contain, and our read handed it the whole ask, so an ask that quotes a sentence never written word for word got no recall at all; a live Letta agent searches with its own keywords. each engine’s row keeps the read as it was run, and the row under it is a check on it rather than another engine: the same store, read without the date parse (on 38 of the 40 people), or without the quotes.
each bracket below is that comparison’s own 95% interval, with no correction for how many were made. counted with Holm’s correction over the 36 comparisons with the four engines, what they will need left out of the count because it carries too little signal in this world, netti-1 read the person better than all four on five families: what they choose, what they do next, what they mean, what changed and when it changed. of those five, only on what they mean does netti-1 also read the person better than that family’s floor, a simple statistic of the person’s own past fitted on the development people; it is level with the floor on what changed, and the floor reads the person better on what they choose, what they do next and when it changed. without the correction, netti-1 is nearer than all four on eight of the ten.
with the correction, netti-1 holds on six against mem0 open source, six against Letta Code, eight against Graphiti (Zep’s open-source engine) and six against Supermemory self-hosted; the other way, two leads hold: mem0 open source on what weighs more and Graphiti (Zep’s open-source engine) on no false insight.
read without its date parse, on the same 38 people, Supermemory self-hosted brought the reader nearer on what they choose, what they do next, what changed and what they will need, and netti-1 stays nearer than it on each. five families read identically.
read without the quotes in its recall query, on the same 40 people, Letta Code brought the reader nearer on what they mean and restraint, and netti-1 stays nearer than it on each. six families read identically.
netti-1 is the lab’s person model, netti-0 version 0.3, under the name it launches with. it is not in innernet yet.
decide · 36 a personof these options, which will they pick?
the pre-registered floor for this family, a count of their own past picks, sits at +0.29 on this scale, nearer than every memory here.
netti-1the othersthe true self, privilegedno memorynats gained over no memory, higher is better
- netti-1the person model (netti-0 v0.3 in the lab), read as text+0.26the line every other row is paired against, person by person
- mem0 open sourcemem0ai 3.3.1, open source: its own extraction, store and search, with our model slot and read+0.15netti-1 nearer by 0.101 [0.079, 0.122] · for 37 of 40 people
- Letta CodeLetta Code 0.34.24, open source: its own memory files, dreaming and recall search, read as our stand-in for a fresh turn+0.10netti-1 nearer by 0.159 [0.129, 0.188] · for 39 of 40 people
- Letta Code, no quotes in recallthe same Letta store and read, the double quotes taken out of its recall query: a check on the row above, not another engine+0.10netti-1 nearer by 0.159 [0.129, 0.188] · for 39 of 40 people
- Graphiti (Zep’s open-source engine)Graphiti 0.31.2, Zep’s open-source engine: its own graph, facts and search, read as its top ten facts+0.03netti-1 nearer by 0.224 [0.198, 0.249] · for 40 of 40 people
- Supermemory self-hostedSupermemory’s self-hosted server 0.0.9, not open source: its own memory agent, store and search, run with no account+0.02netti-1 nearer by 0.234 [0.208, 0.259] · for 40 of 40 people
- Supermemory self-hosted, no date filterthe same Supermemory store and search, the ask read without the engine’s date parse: a check on the row above, not another engine+0.10netti-1 nearer by 0.150 [0.127, 0.173] · for 38 of 38 people
- flat factsbuilt like mem0: our build of its published design+0.07netti-1 nearer by 0.185 [0.152, 0.219] · for 38 of 40 people
- temporal graphbuilt like zep: our build of its published design+0.09netti-1 nearer by 0.168 [0.140, 0.196] · for 38 of 40 people
- memory blocksbuilt like letta: our build of its published design+0.10netti-1 nearer by 0.152 [0.126, 0.178] · for 38 of 40 people
- retrievalinnernet’s retrieval pipeline over the raw stream, on a local hashed embedder since a provider outage+0.21netti-1 nearer by 0.045 [0.024, 0.065] · for 31 of 40 people
- the true selfprivileged: the hidden self the world was built from, as text+0.10
after one forget, the forgotten words were served again to
of 40 people (38 for Supermemory self-hosted, no date filter). served again: the forgotten words turned up in a context handed to the reader after the forget night, found by the world’s own token match, the same check for every row. still stored: the same match on everything the memory keeps,netti-1’s learned state, checkpoints, model-call cache and row log included, and mem0’s change log. where an engine keeps no link from what it derived to the row it came from, the forget is ours, and its row says so.
served tokens a read, the mean
- bench
- likeness-1.0.0, pre-registered; this run came after the reveal
- people
- 40 synthetic test people
- run
- 2026-10-26
- reader
- Claude Haiku 5.5 for every arm
- netti-1
- version 0.3: the frozen model with a forget that keeps nothing and a few more rows read beside it, tuned on development people only
- mem0 open source
- mem0ai 3.3.1: its own extraction, store and search; Claude Haiku 5.5 in its model slot (its default is gpt-5-mini), its own local embedder, read as its top twenty memories for the ask
- Letta Code
- Letta Code 0.34.24, open source, on its local backend: its own memory files, dreaming and recall search; a 100,000-token registered window, nine tools taken out, no persona; read as its core memory and a recall search for the whole ask, our stand-in for a fresh turn, without its recall agent
- no quotes in recall
- the same Letta store and read, each double quote in the recall query made a space; rebuilt from the kept transcript at no cost, after the same rebuild with the original query had reproduced every stored read
- Graphiti
- Graphiti 0.31.2, Zep’s open-source engine, on an embedded graph store, with a small local embedder in place of its default: its own prompts, graph and search, read as its top ten facts with their dates, invalidated ones served too by design, without entity summaries or its node-distance and reranked searches; not Zep’s cloud product
- Supermemory self-hosted
- Supermemory’s self-hosted server 0.0.9, not open source, with no account: its memory agent, store and search, dreaming at once on each day (its default batches), read through MemoryBench’s own search call; not the hosted product
- no date filter
- the same Supermemory store, with the ask’s first space made a line break, which the engine’s own rule exempts from its date parse; logged during the run at no cost
- the engines
- Claude Haiku 5.5 in every engine’s model slot (thinking off), local embedders, one write a day, the same served budget; each event’s time, port and role, not its privacy tier or which event it corrects
- reads
- netti-1 and retrieval read with the answer options; netti-1 makes a read-time model call on 615 reads (replayed from its cache here), retrieval pays $1.38 for query expansion; the engines search with the ask alone
- private leaks
- 8 tempting tasks a person, apart from the log loss: the private fact in the packet handed to the reader, then in the reader’s draft. netti-1 0 and 0 of 320, mem0 open source 295 and 3 of 320, Letta Code 266 and 0 of 320, Letta Code, no quotes in recall 266 and 0 of 320, Graphiti (Zep’s open-source engine) 270 and 5 of 320, Supermemory self-hosted 212 and 2 of 320, Supermemory self-hosted, no date filter 202 and 2 of 304, flat facts 245 and 2 of 320, temporal graph 286 and 5 of 320, memory blocks 312 and 4 of 320, retrieval 316 and 5 of 320. the engines and retrieval were not given the privacy tiers, so their packets say what they served without them, not what they would withhold
- built like
- flat facts, a temporal graph and memory blocks with a sleep-time rewrite: our builds of mem0’s, zep’s and letta’s published designs, with our prompts and a local hashed embedder, never their products
- paired
- per person: Δ in nats, 10,000 resamples of people, seed 1729, 95% percentile interval for each comparison; Holm’s correction over the 36 comparisons with the four engines, what they will need left out, from paired t tests
- files
- docs/research/person-models/field, the v6 files: score-raw-test-v6.txt, v6-paired.json, v6-forget.json, v6-economy.json, v6-twins.json, v6-nodate.json, v6-noquote.json, v6-letta-echo.json, v6-leak.json; the Holm counts and the floor tests: packages/bench/person/out/analyze11/v6-holm-1.1-counts.json
the raw expected log loss every gain is taken from, lower is better
| arm | decide | continue | intend | keep.track | keep.phase | keep.source | keep.need | restraint | know | reveal |
|---|---|---|---|---|---|---|---|---|---|---|
| netti-1 | 1.198 | 2.243 | 1.071 | 0.577 | 0.682 | 0.740 | 2.009 | 0.183 | 0.617 | 1.997 |
| mem0 open source | 1.299 | 2.360 | 1.527 | 0.966 | 0.942 | 0.942 | 2.211 | 0.267 | 0.508 | 2.002 |
| Letta Code | 1.357 | 2.546 | 1.581 | 1.183 | 0.914 | 0.865 | 2.092 | 0.387 | 0.550 | 1.931 |
| Letta Code, no quotes in recall | 1.357 | 2.546 | 1.506 | 1.183 | 0.919 | 0.904 | 2.092 | 0.261 | 0.550 | 1.931 |
| Graphiti (Zep’s open-source engine) | 1.422 | 2.594 | 1.647 | 1.237 | 1.009 | 1.030 | 2.251 | 0.429 | 0.702 | 1.794 |
| Supermemory self-hosted | 1.432 | 2.587 | 1.622 | 1.234 | 1.100 | 0.904 | 2.227 | 0.259 | 0.562 | 2.025 |
| Supermemory self-hosted, no date filter | 1.352 | 2.497 | 1.666 | 1.123 | 1.095 | 0.897 | 2.144 | 0.259 | 0.557 | 2.024 |
| flat facts | 1.383 | 2.499 | 1.623 | 1.225 | 1.131 | 1.120 | 2.155 | 0.323 | 0.566 | 1.962 |
| temporal graph | 1.367 | 2.529 | 1.635 | 1.244 | 1.118 | 1.064 | 2.184 | 0.357 | 0.553 | 1.951 |
| memory blocks | 1.351 | 2.530 | 1.391 | 0.992 | 0.988 | 1.133 | 2.059 | 0.285 | 0.611 | 1.865 |
| retrieval | 1.243 | 2.492 | 1.103 | 0.781 | 0.727 | 0.674 | 2.119 | 0.165 | 0.485 | 1.766 |
| the true self | 1.353 | 2.252 | 0.956 | 0.176 | 0.105 | 1.122 | 2.017 | 0.346 | 0.557 | 0.554 |
| no memory | 1.453 | 2.593 | 1.603 | 1.387 | 1.386 | 1.085 | 2.079 | 0.374 | 0.693 | 1.608 |
limits
what it cannot tell you.
- the people are programs. every number here is about synthetic people of this kind, and the claims are ordinal and regime-qualified: one memory beats another in worlds like these. none of it is an estimate of how well anything understands a human.
- the text is templated, so it is cleaner and more regular than a real life. a second world with different phrasing and different attributes is the honest next test.
- the same team built the world and the model it scores. a held-out decision rule, public randomness for the seed and a privileged learner beside every number reduce that problem; a world built by someone else would remove more of it.
- 40 test people detect differences of about 0.025 nats per decision. smaller ones are intervals, not findings. the confirmatory chain was scored on them once; every comparison since, the field among them, reuses the same people and is exploratory.
- one reader carries most of it: Claude Haiku 5.5. the cross-reader study rests on 10 people and Claude readers only.
- the labels are given, and not every memory got them. who wrote each event and how private it is arrive with the world, and netti-1 reads both. the vendors’ engines and retrieval got each event’s time, port and role, and a label on a correction, but not its privacy tier or which earlier event it corrects. a memory that had to infer them would face a harder test.
- the field comparison came after the reveal, outside the confirmatory chain, and raw. the built-like memories are our builds of published designs, not the products, and they miss in both directions: the real mem0 package did better than the memory built like it on 6 of the 10 families and worse on 1, Letta Code did better than the memory built like it on 2 of the 10 families and worse on 3, Graphiti did better than the memory built like it on 1 of the 10 families and worse on 4, each with intervals clear of zero; Letta Code with the quotes out of its recall query did better than the memory built like it on 2 of the 10 families and worse on 2.
- none of the vendors’ engines ran as its vendor ships it end to end. each ran its own code with Claude Haiku 5.5 in its model slot, a local embedder, one write a day and our read, and none is a hosted product: Graphiti is not Zep’s cloud, and Supermemory’s self-hosted server is not its hosted one. Letta’s read is our stand-in for a fresh turn, its core memory and a recall search with the whole ask, without its recall agent; Graphiti’s is its top ten facts, without entity summaries or its reranked search; Supermemory dreamed at once on each day, where its default waits and batches.
- who wrote each forget: netti-1’s and Graphiti’s are their own; mem0, Letta Code and Supermemory keep no link from what they derive to the row it came from, so their forget is ours, deleting derived text near the row. Letta’s own replies and summaries were left out of that rule, and of the 104 leaking lines it wrote by the forget day, four were near enough for the rule to take: every one of the 31 people they reached would still have been served.
- what synthetic people cannot settle: whether a person recognises themselves in a person model, whether they would want it kept, and the minutes or money it saves them.