labsbenchmarks

innernet labsmeasured · 2026-10-27

memory, measured.

every benchmark we have run, the line each one was measured against, the same reader on other memories beside it, and exactly how it was made. each test run once, the misses kept in the same table.

92.0%longmemeval-s, all 500 questions and the 350 it was not tuned on. the leaderboard door, not innernet as shippedinnernet as shipped, on memoryagentbench's year of chats, where this door reads 85.7 (current main at 1e9b3606): 17.0%the same on main at 9949e7ec (24 oct), within noise of it: 15.7%
85.5memoryagentbench, accurate retrieval average. the leaderboard door, not innernet as shipped
54.0memoryagentbench, all four kinds averaged the paper's way, through the same doorplain bm25 as the paper set it, the same reader: 52.5gpt-5-mini reading the whole context, as published: 60.6
5 of 9things netti-1 reads people better on than each of four vendors' own memory engines (mem0, letta code and zep's graphiti, open source; supermemory's self-hosted server), in likeness, after a correction for the 36 comparisons, of the nine a headline may name. on only one of the five, what they mean, does it also beat that family's floor, a simple statistic of the person's own past. exploratory, and not what the product runs today

every number for innernet's memory on this page is read by claude haiku 5.5. longmemeval-s, LME(S*) and the book summaries are judged by claude sonnet 5.5 with each benchmark's own prompts; the rest are scored by each harness's own rule, named on its row.

every test

  • recallin the field

    longmemeval-s

    500 questions, each about a history of about 115k tokens: what was said, what changed, when, and what was never said.

    92.0%95% interval 89.3 to 94.1
    • the leaderboard door · claude haiku 5.592.0
    • gpt-4o reading the whole history (the paper)60.6
    • supermemory in production, self-reported, up to (81.6 with gpt-4o reading, 85.2 with gemini-3-pro)85.2
    • zep 2026, self-reported (gpt-5.4 reads and judges)90.2
    • honcho, self-reported (claude haiku 4.5 reads, gpt-4o judges)90.4
    • hindsight, self-reported (gemini-3-pro reads, gpt-oss-120b judges)91.4
    • memmachine, self-reported (gpt-5-mini reads, gpt-4o-mini judges)93.0
    • mem0 platform, self-reported (gpt-4o by default; 94.4 at top 200)94.8
    • mastra om, self-reported (gpt-5-mini reads; 94.87)94.9
    • supermemory, self-reported (gpt-4o reads and judges, 20 merged slots)97.0

    the same reader, run by us claude haiku 5.5 in front of each, so only the memory moves.

    • 90.0no memory system: haiku reads the whole history itself100 held-out questions, 16 or 17 of each type; interval 82.6 to 94.5; 172,871 prompt tokens a question by the api's count

    with the same reader and judge, haiku reading the whole history itself scores 90.0 on 100 held-out questions; on those same 100 the door reads 87, a paired 2 against 5 (p 0.45): level with reading everything, not above it. the published lines are other readers and other judges, almost all reported by the people who built them, so none of them is a head-to-head.

    • single-session, assistant56 of 56
    • single-session, user68 of 70
    • knowledge update74 of 78
    • temporal reasoning125 of 133
    • multi-session112 of 133
    • preference25 of 30
    • abstention (inside the types above)24 of 30
    data
    LongMemEval-S, the cleaned release (xiaowu0162/LongMemEval, MIT)
    run
    2026-10-21, all 500 questions, once
    tuning
    the door and its answer step were tuned on a fixed dev split of 150 of these 500 (30% of each type, chosen by a hash of the id); on the other 350 it reads 92.0 (322 of 350), on the 150 92.0 (138 of 150)
    memory
    the leaderboard door, no embeddings
    reader
    claude haiku 5.5, the door's own answer step
    judge
    claude sonnet 5.5 with LongMemEval's own judge prompts (the paper used gpt-4o)
    read
    about 21k tokens of memory per answer (top 100 items)
    published
    the vendor lines as each vendor reported them by 8 oct 2026, readers and judges named; none is like-for-like with another (packages/bench/BENCHMARKS.md, section 3)
    same reader
    2026-10-16, the whole history in the prompt and the same answer rules, on 100 of the held-out 350, balanced by type; the door's 87 on those 100 is this run's own rows for them
    cost
    $3.76 for the run
    file
    packages/bench/results-aml-lmes-noembed-haiku.json; the whole history, packages/bench/results-infinite-window-lmes-haiku-5-5.json
    • graded by Claude with the official prompts, not by gpt-4o, so it does not line up exactly with numbers graded another way
    • most other numbers on this test are reported by the people who built the system, with their own readers and judges, so the ticks are context, not a ranking
    • the history fits in today's long context windows, so the test partly measures how well a memory uses its budget: the same reader with the whole history in front of it is level with the door
    • this is the leaderboard door, not innernet as shipped: the shipped path read through innernet_context answered 15.7% of memoryagentbench's year of chats on main at 9949e7ec (24 oct) and 17.0% on current main at 1e9b3606, within noise of each other (that row, below), so this is not the product's number

  • recallmeasured

    longmemeval-s in supermemory's harness

    the same 500 questions, run in MemoryBench, the open harness supermemory publishes, with its own prompts.

    91.8%
    • the leaderboard door · claude haiku 5.591.8
    • supermemory, self-reported in this harness (gpt-4o reads and judges, 20 merged slots)97.0

    the same reader, run by us claude haiku 5.5 in front of each, so only the memory moves.

    • 85.8the door with 20 items instead of 100, supermemory's slot count4,023 tokens a read by the harness's own count, against 21,580 at 100

    with the same reader and judge, the door held to supermemory's 20 slots reads 85.8: twenty raw items lose the breadth a counting question needs, and multi-session falls to 67.7.

    harness
    supermemoryai/memorybench at 063b86b, the leaderboard door added as a provider (packages/bench/memorybench)
    run
    2026-10-21, all 500 questions; the door was tuned on 150 of them (the longmemeval-s row says how)
    reader
    claude haiku 5.5, no extended thinking
    judge
    claude sonnet 5.5 with MemoryBench's own judge prompts
    read
    21,580 tokens per answer at 100 items, 4,023 at 20; search 2.2s and 1.0s
    file
    packages/bench/results-memorybench-lme-500-innernet-haiku.json; at 20 items, results-memorybench-lme-500-innernet-haiku-k20.json
    • supermemory's own 97.0 used gpt-4o to answer and to judge, so the two are not a head-to-head until both run under one reader and one judge
    • the leaderboard door, not innernet as shipped

  • retrievalthe paper's averageabove the published line, with a 2026 reader

    accurate retrieval, the four averaged

    memoryagentbench's first kind, added up as its paper adds it: the mean of the four rows below, one fact in 197k, a year of chats, a novel, and facts in a chain.

    85.5strict reading by hand 84.2
    • the leaderboard door · claude haiku 5.585.5
    • best in the paper's table (gpt-5-mini, long context)74.4
    • best memory agent in the table (hipporag-v2)65.1
    • bm25 in the table, gpt-4o-mini reading60.5
    • mem0 in the table32.6

    11.1 above the published line it is measured against, with a 2026 reader.

    the same reader, run by us claude haiku 5.5 in front of each, so only the memory moves.

    • 21.7no memory, the question alonemostly the reader saying it has nothing to go onstrict 18.4
    • 64.6bm25, the harness's own, 512-token chunks, top 10strict 52.3
    • 66.3bm25 as the paper set it: 512-token chunks on the three synthetic sets, 4,096 on the novelsstrict 54.1
    • 70.8the same bm25, 4,096-token chunks on all fourstrict 55.0
    • 52.2a small embedder (bge-small) on the same chunks, top 10strict 41.5
    • 82.3the door held to the read budget of bm25 on 512-token chunksstrict 62.3
    • 83.1the door without its query expansionstrict 63.3

    with the same reader, the harness's bm25 averages 64.6 on 512-token chunks, 66.3 as the paper set it and 70.8 on 4,096-token chunks; the door held to the 512-chunk read budget averages 82.3, so most of the gap is the door, not the budget. that budget counts the tokens read: the prompt billed runs 0.3 to 5.8% higher, and the door adds one haiku query expansion a question and one extraction a passage stored. a cell's strict is one string rule for every arm (the gold, or the gold within three more words and no second candidate): by it the door reads 62.6, where each pass read by hand gives the 84.2 under the score.

    how
    the mean of SH-QA, MH-QA, LME(S*) and EventQA, each weighed the same, as the paper's AR-avg column is
    published
    MemoryAgentBench v4 (arXiv 2507.05257v4, 28 jun 2026), table 3; its memory agents run on gpt-4o-mini unless named
    reader
    claude haiku 5.5 on all four
    strict
    84.2 under the score is the four rows read by hand, every pass that hedges or contradicts itself marked wrong (LME(S*) is judged, so its graded score stands), then averaged. the same-reader cells use the controls file's string rule, one rule for every arm, under which the door reads 62.6
    bm25 as the paper set it
    the paper ran bm25 on 512-token chunks for SH-QA, MH-QA and LME(S*) and on 4,096 for EventQA and its other tasks (section 4.1, appendix F.3): 71, 55 and 59 from the 512 arm and 80.2 from the 4,096 arm, averaged
    budget
    the door at bm25's budget takes its top 52, 61, 30 and 30, chosen on each set's mean tokens read (characters ÷ 4), never on score
    files
    packages/bench/results-mab-door-haiku-5-5.json; the same-reader rows, packages/bench/results-mab-controls-haiku-5-5.json
    • an average of four tests with three scorers (substring match on three, an llm judge on one), which is the paper's own way of adding them
    • no memory mostly records the reader saying it has nothing to go on, on 89 of 100 SH-QA questions, 73 of 100 MH-QA and all 300 LME(S*); only the novels' 51.4 is more than that: what the reader infers from the story so far and knows of the books
    • 92 of LME(S*)'s 300 come from the dev split the door was tuned on (the longmemeval-s row says how); on the other 208 it reads 86.1
    • the door's rows ran on 22 oct and the same-reader rows on 25 oct, separate runs of a reader that cannot be made fully repeatable
    • the reader is a 2026 model; the paper's memory agents ran on gpt-4o-mini and its long-context rows are 2025 models

  • retrievalabove the published line, with a 2026 reader

    one fact in 197k tokens

    memoryagentbench SH-QA (RULER-QA1): 100 questions, each answered by one passage hidden in 197k tokens of documents.

    94.095% interval 87.5 to 97.2strict reading by hand 91.0
    • the leaderboard door · claude haiku 5.594.0
    • best in the paper's table (gemini-2.0-flash, long context)87.0
    • gpt-5-mini, long context85.0
    • best memory agent in the table (hipporag-v2)76.0
    • bm25 in the table, gpt-4o-mini reading66.0
    • mem0 in the table25.0

    7.0 above the published line it is measured against, with a 2026 reader.

    the same reader, run by us claude haiku 5.5 in front of each, so only the memory moves.

    • 9.0no memory, the question aloneit said it had nothing to go on in 89 of 100strict 5.0
    • 71.0bm25, the harness's own, 512-token chunks, top 10, as the paper set it herestrict 60.0 · 6,422 tokens a read
    • 69.0the same bm25, 4,096-token chunksstrict 58.0 · 49,607 tokens a read
    • 53.0a small embedder (bge-small) on the same chunks, top 10strict 45.0 · 6,391 tokens a read
    • 95.0the door held to bm25's token budgetits top 52strict 79.0 · 6,379 tokens a read
    • 93.0the door without its query expansionstrict 77.0 · 13,032 tokens a read

    a cell's strict is one string rule for every arm: the answer is the gold, or holds it within three more words with no second candidate. by it the door reads 74, where each pass read by hand, only the hedged ones marked wrong, gives the 91.0 under the score; the string rule also fails right answers said at length.

    data
    ai-hyz/MemoryAgentBench (MIT), ruler_qa1_197K
    harness
    HUST-AI-HYZ/MemoryAgentBench at 5380260: its prompts, its reader layout, its retrieval query, its substring exact match
    published
    MemoryAgentBench v4 (arXiv 2507.05257v4), table 3
    run
    2026-10-22, piloted on 20 questions, then all 100 once
    reader
    claude haiku 5.5 (the paper's memory agents ran on gpt-4o-mini)
    read
    12.4k tokens per answer
    same reader
    2026-10-25, all 100 per arm: the harness's bm25 agent (rank_bm25 on its own chunks, top 10), a local bge-small-en-v1.5 on the same chunks, and the door with one part changed
    files
    packages/bench/results-mab-door-haiku-5-5.json; the same-reader rows, packages/bench/results-mab-controls-haiku-5-5.json
    • substring match accepts a long answer that contains the right one; marking hedged answers wrong gives 91.0
    • the reader is a 2026 model; the paper's rows are 2025 models

  • retrievalabove the published line, with a 2026 reader

    a year of chats, 355k tokens

    memoryagentbench LME(S*): five long histories merged from LongMemEval, 60 questions each.

    85.795% interval 81.2 to 89.2
    • the leaderboard door · claude haiku 5.585.7
    • best in the paper's table (gpt-5-mini, long context)63.3
    • best memory agent in the table (mirix on gpt-4.1-mini)51.0
    • bm25 in the table, gpt-4o-mini reading45.3
    • mem0 in the table36.0

    22.4 above the published line it is measured against, with a 2026 reader.

    the same reader, run by us claude haiku 5.5 in front of each, so only the memory moves.

    • 4.3no memory: the answer step over an empty memoryit said it had nothing to go on in all 300; its 13 right are the 13 questions whose answer is that it was never saidstrict 0.0
    • 59.0bm25, the harness's own, 512-token chunks, top 10, as the paper set it herestrict 28.0 · 5,949 tokens a read
    • 72.0the same bm25, 4,096-token chunksstrict 31.7 · 46,277 tokens a read
    • 46.7a small embedder (bge-small) on the same chunks, top 10strict 17.7 · 5,807 tokens a read
    • 82.7the door held to bm25's token budgetits top 30strict 28.7 · 5,975 tokens a read
    • 79.3the door without its query expansion6.3 under the door at a fixed top 100 (27 against 8, p 0.0019), reading a third fewer tokens; expansion at equal tokens was not runstrict 28.3 · 12,870 tokens a read
    • 15.7innernet as shipped, main at 9949e7ec (24 oct): its own write path, read through innernet_contextinterval 12.0 to 20.2; the same 15.7 for an app cleared to read what is personal1,936 tokens a read
    • 9.7the same, the build of 11 oct8.7 for an app cleared to read what is personal971 tokens a read
    • 17.0the same, current main at 1e9b3606, after #332 changed the readinterval 13.2 to 21.7; the same 17.0 for an app cleared to read what is personal, at 1,917 tokens a read (characters ÷ 4, as every cell); paired against 9949e7ec, not significant (p 0.61, and 0.57 at personal depth)1,903 tokens a read

    the door above is the leaderboard door, a research memory built for benches like this. with the same reader, plain bm25 on 4,096-token chunks already clears the published line (72.0 against 63.3), and the door is 13.7 over it (58 against 17, p 2e-6). innernet as shipped, written and read the way a connected ai reads it, answers 17.0% of the same 300 at both depths on current main at 1e9b3606, 51 questions. main at 9949e7ec answers 15.7, 47 questions, up from 9.7% in the build of 11 oct; paired on the same 300, current main against it is within noise (p 0.61), so it is not a measured gain. the shipped door is not the leaderboard door. on current main it still loses the answer in the same three places: it sends at most five facts, chosen because they share one word with the ask; for someone whose memory is only personal, its focus is the same few sessions whatever is asked; and what the assistant said is not kept when memory is written. strict here is a string rule over free-text answers that the judge reads, so it fails most right answers; by it the door reads 30.

    • single-session, assistant30 of 30
    • knowledge update42 of 45
    • single-session, user41 of 45
    • temporal reasoning64 of 75
    • multi-session58 of 75
    • preference22 of 30
    data
    ai-hyz/MemoryAgentBench, longmemeval_s* (300 questions)
    published
    MemoryAgentBench v4 (arXiv 2507.05257v4), table 3
    run
    2026-10-22, piloted on 20, then all 300 once
    reader
    claude haiku 5.5, the door's own answer step, in every arm
    judge
    claude sonnet 5.5 with LongMemEval's own prompts (the paper used gpt-4o)
    read
    19.1k tokens per answer
    tuning
    these 300 are LongMemEval-S questions, and 92 of them are in the 150 the door and its answer step were tuned on (the longmemeval-s row says how); on the other 208 the door reads 86.1 (179 of 208), on the 92 84.8 (78 of 92)
    same reader
    2026-10-25, all 300 per arm, each searched with the door's own question
    as shipped
    lib/self, lib/mcp and packages/core bundled from current main at 1e9b3606 (after #332 and #333, run 25 oct utc), from main at 9949e7ec (committed 24 oct, run 25 oct), and at a33005c5 (11 oct), on production's memDb and a simulated clock: every session captured through the auto-capture door, each question read through innernet_context at standard depth (what any connected app gets) and at personal depth, answered and judged as above
    bench fake
    since #332 the personal fold claims a lease and reads its note queue with a filter the bench's stand-in database could not read, so for 1e9b3606 the stand-in answers both as migration 163 does. it is bench code, not product code; a $0 dry run of the 9949e7ec bundle is byte-identical before and after the change
    current main against 9949e7ec
    the same 300, paired: 1e9b3606 is right on 19 that 9949e7ec missed and misses 15 it had (exact p 0.61); at personal depth 16 and 12 (p 0.57). the writes were paid for again, so this mixes #332's changes with run-to-run variation in the writes
    9949e7ec against 11 oct
    the same 300, paired: 9949e7ec is right on 29 the older build missed and misses 11 it had (exact p 0.006); at personal depth 31 and 10 (p 0.0015)
    tokens
    every cell counts characters ÷ 4; the shipped results files count characters ÷ 3, the person bench's rule (2,538 a read on 1e9b3606, 2,582 on 9949e7ec, 1,295 on 11 oct), shown here ÷ 4 from the same reads
    files
    packages/bench/results-mab-door-haiku-5-5.json; the same-reader rows, packages/bench/results-mab-controls-haiku-5-5.json; as shipped, packages/bench/results-mab-today-lme-1e9b3606-haiku-5-5.json (current main), results-mab-today-lme-main-haiku-5-5.json (9949e7ec) and results-mab-today-lme-haiku-5-5.json (11 oct)
    • judged by Claude, not by gpt-4o
    • the shipped bundle runs without embedding keys, so its shortlist is by words; every read here sent the whole small map, so this likely changes little, but it is not production's exact read
    • the door's query expansion is measured at a fixed top 100, where taking it away also takes away tokens (19,100 a read against 12,870). held to 5,975 tokens, the door with expansion is still 3.3 above the door without it reading 12,870 (18 against 8, p 0.076, not significant); expansion at equal tokens was not run, and the door's rows (22 oct) and these (25 oct) are separate runs
    • 92 of these 300 are in the dev split the door was tuned on; on the other 208 it reads 86.1

  • retrievalabove the published line, with a 2026 reader

    a novel: what happens next

    memoryagentbench EventQA: five novels of about 534k tokens, 100 questions each, six possible next events to choose from.

    89.295% interval 86.2 to 91.6strict reading by hand 89.2
    • the leaderboard door · claude haiku 5.589.2
    • best in the paper's table (gpt-4.1-mini, long context)82.6
    • gpt-5-mini, long context78.2
    • best memory agent in the table (bm25)74.6
    • mem0 in the table37.5

    6.6 above the published line it is measured against, with a 2026 reader.

    the same reader, run by us claude haiku 5.5 in front of each, so only the memory moves.

    • 51.4no memory, the question alonechance is 16.7; the one floor here that is more than the reader declining: what it infers from the story so far, and what the reader knows of the novels
    • 73.2bm25, the harness's own, 512-token chunks, top 105,908 tokens a read
    • 80.2the same bm25, 4,096-token chunks, as the paper set it here44,501 tokens a read
    • 67.2a small embedder (bge-small) on the same chunks, top 105,765 tokens a read
    • 82.6the door held to the read budget of bm25 on 512-token chunksits top 305,917 tokens a read
    • 89.0the door without its query expansion24,577 tokens a read

    with no book at all, only the question with the story so far and its six options, the reader picks the right next event 51.4 times in a hundred, against 16.7 by chance: the floor here is high, what the reader can infer from the story so far and what it knows of these novels. bm25 as the paper set it, on 4,096-token chunks, reads 80.2, and the door is 9.0 over it (68 against 23, p 3e-6). held to the read budget of bm25 on 512-token chunks, a budget the paper did not use here, the door falls to 82.6: level with bm25 on 4,096-token chunks (54 against 42, p 0.26) at about an eighth of its tokens, and above bm25 on 512 (75 against 28, p 4e-6). every answer names one option, so strict reads the same.

    data
    ai-hyz/MemoryAgentBench, eventqa_full (500 questions)
    published
    MemoryAgentBench v4 (arXiv 2507.05257v4), table 3
    run
    2026-10-22, piloted on 20, then all 500 once
    book
    cut into passages of 120 to 300 words, each marked with its place in the book
    reader
    claude haiku 5.5
    read
    19.4k tokens per answer
    same reader
    2026-10-25, all 500 per arm
    files
    packages/bench/results-mab-door-haiku-5-5.json; the same-reader rows, packages/bench/results-mab-controls-haiku-5-5.json
    • the passage cut is ours; the books ship as one paragraph per line
    • every pass names the single option it chose, so the strict reading is the same 89.2

  • retrievalunder the published line, inside its interval

    facts in a chain, 421k tokens

    memoryagentbench MH-QA (RULER-QA2): 100 questions that need two documents put together, in 421k tokens.

    73.095% interval 63.6 to 80.7strict reading by hand 71.0
    • the leaderboard door · claude haiku 5.573.0
    • best in the paper's table (mirix on gpt-4.1-mini)75.0
    • best long context in the table (gpt-5-mini)71.0
    • bm25 in the table, gpt-4o-mini reading56.0
    • mem0 in the table32.0

    2.0 under the published line it is measured against.

    the same reader, run by us claude haiku 5.5 in front of each, so only the memory moves.

    • 22.0no memory, the question aloneit said it had nothing to go on in 73 of 100strict 17.0
    • 55.0bm25, the harness's own, 512-token chunks, top 10, as the paper set it herestrict 48.0 · 5,889 tokens a read
    • 62.0the same bm25, 4,096-token chunksstrict 50.0 · 44,940 tokens a read
    • 42.0a small embedder (bge-small) on the same chunks, top 10strict 36.0 · 5,815 tokens a read
    • 69.0the door held to bm25's token budgetits top 61strict 59.0 · 5,901 tokens a read
    • 71.0the door without its query expansionstrict 59.0 · 10,122 tokens a read

    a cell's strict is one string rule for every arm: the answer is the gold, or holds it within three more words with no second candidate. by it the door reads 57, where each pass read by hand, only the self-contradicting ones marked wrong, gives the 71.0 under the score.

    data
    ai-hyz/MemoryAgentBench, ruler_qa2_421K
    published
    MemoryAgentBench v4 (arXiv 2507.05257v4), table 3
    run
    2026-10-22, piloted on 20, then all 100 once
    reader
    claude haiku 5.5
    read
    9.8k tokens per answer
    same reader
    2026-10-25, all 100 per arm
    files
    packages/bench/results-mab-door-haiku-5-5.json; the same-reader rows, packages/bench/results-mab-controls-haiku-5-5.json
    • two points under the best row, inside the interval; most misses are answers that drop part of the gold span
    • a strict reading of self-contradicting answers gives 71.0

  • changethe paper's averagebuilt for it

    selective forgetting, the two averaged

    memoryagentbench's fact consolidation, added up as its paper adds it: the mean of the two rows below, single-hop and through a chain.

    78.0through the general door: 33.5, strict 33.0
    • phases, a read path built for this test · claude haiku 5.578.0
    • arXiv 2606.01435's two, averaged: a 2026 workshop poster with a pipeline built for this task (gpt-4o)67.0
    • best in the benchmark's own table (gpt-5-mini, long context)53.0
    • best memory agent in the table (hipporag-v2)29.5
    • bm25 in the table, gpt-4o-mini reading25.5
    • mem0 in the table10.0

    11.0 above the published line it is measured against, with a 2026 reader. through the general door, 33.5 under it.

    the same reader, run by us claude haiku 5.5 in front of each; the two thinking cells also change how it reads, so there the reader moves too.

    • 38.5bm25, the top 10 fact lines
    • 59.5no memory system: haiku reads all 262k, thinking on
    • 52.5the general door, thinking onstrict 51.5

    through the general door, with nothing in it built for this test, the two average 33.5: under the benchmark's own gpt-5-mini line of 53.0, and under the same reader reading all 262k itself (59.5). the 78.0 is phases, a read path built for fact consolidation.

    how
    the mean of FC-SH and FC-MH at 262k, each weighed the same
    ours
    phases v1.1, a read path built for this test; the general door, with nothing built for it, sits beside it
    strict
    the general door's strict is the fc-door file's rule: the answer names one candidate
    published
    MemoryAgentBench v4 (arXiv 2507.05257v4), table 3; arXiv 2606.01435 for the 93 and 41
    reader
    claude haiku 5.5
    files
    packages/bench/results-mab-fc-haiku-5-5.json; the general door, packages/bench/results-mab-fc-door-haiku-5-5.json
    • the 78.0 is a read path built for fact consolidation, and so is the 67.0 it is measured against; the general door's 33.5 is the number to set beside the paper's table
    • the same-reader rows and the general door are the two rows' own results averaged

  • changebuilt for it

    the newest fact, single-hop, 262k

    memoryagentbench FactConsolidation: a pool of facts where a later fact replaces an earlier one; 100 questions on the current value.

    9495% interval 87.5 to 97.2through the general door: 66, strict 65
    • phases, a read path built for this test · claude haiku 5.594
    • arXiv 2606.01435, a 2026 workshop poster with a pipeline built for this task (gpt-4o)93
    • best in the benchmark's own table (gpt-5-mini, long context)78
    • best memory agent in the table (hipporag-v2)54
    • bm25 in the table, gpt-4o-mini reading48
    • mem0 in the table18

    1.0 above the published line it is measured against, with a 2026 reader. through the general door, 27.0 under it.

    the same reader, run by us claude haiku 5.5 in front of each; the two thinking cells also change how it reads, so there the reader moves too.

    • 74bm25, the top 10 fact lines
    • 86no memory system: haiku reads all 262k, thinking on
    • 93the general door, thinking on1,881 tokens a read

    through the general door the same reader answers 66, under bm25 over the fact lines (74) and under the benchmark's own gpt-5-mini line (78); with thinking on it reaches 93. the 94 is phases, a read path built for this test.

    data
    ai-hyz/MemoryAgentBench, factconsolidation_sh_262k
    published
    MemoryAgentBench v4 (arXiv 2507.05257v4), table 3; arXiv 2606.01435
    run
    2026-10-21; the read path tuned on the 6k cells, 32k to 262k held out for version 1, which scored 95 here; version 1.1 fixed two bugs that run exposed and re-ran every cell, so its 94 is not a held-out number
    memory
    phases v1.1, write-time phases: a later value for the same subject retires the earlier one, which stays in history
    general door
    2026-10-25, the same 100 questions through the leaderboard door, nothing in it built for this test: 66 (95% interval 56.3 to 74.5) with the harness's 16-token answer, 93 with thinking on
    strict
    the general door's strict is the fc-door file's rule: the answer names one candidate
    reader
    claude haiku 5.5
    files
    packages/bench/results-mab-fc-haiku-5-5.json; the general door, packages/bench/results-mab-fc-door-haiku-5-5.json
    • phases is a read path built for this test; the general door is the one to set beside the benchmark's own table
    • level with arXiv 2606.01435, within noise
    • version 1.1 fixed two bugs the held-out run exposed and re-ran every cell; version 1 scored 95

  • changebuilt for it

    the newest fact, through a chain, 262k

    the same pool, with questions that follow two or three facts in a row, each of which may have been replaced.

    6295% interval 52.2 to 70.9through the general door: 1
    • phases, a read path built for this test · claude haiku 5.562
    • arXiv 2606.01435, a 2026 workshop poster with a pipeline built for this task (gpt-4o)41
    • best in the benchmark's own table (gpt-5-mini, long context)28
    • best memory agent in the table (contriever and memorag)7
    • bm25 in the table, gpt-4o-mini reading3
    • mem0 in the table2

    21.0 above the published line it is measured against, with a 2026 reader. through the general door, 40.0 under it.

    the same reader, run by us claude haiku 5.5 in front of each; the two thinking cells also change how it reads, so there the reader moves too.

    • 3bm25, the top 10 fact lines
    • 33no memory system: haiku reads all 262k, thinking on
    • 12the general door, thinking onstrict 10 · 1,917 tokens a read

    through the general door the same reader answers 1 in 100, and bm25 over the fact lines 3. with thinking on the door reaches 12, and the reader with all 262k in front of it 33. the 62 is phases, a read path built for this test.

    data
    ai-hyz/MemoryAgentBench, factconsolidation_mh_262k
    published
    MemoryAgentBench v4 (arXiv 2507.05257v4), table 3; arXiv 2606.01435
    run
    2026-10-21; the read path tuned on the 6k cells, 262k held out for version 1, which scored 61 here; version 1.1 fixed two bugs that run exposed and re-ran every cell, so its 62 is not a held-out number
    read path
    phases v1.1: current values only, one subject at a time, following the chain
    general door
    2026-10-25, the same 100 questions through the leaderboard door, nothing in it built for this test: 1 (95% interval 0.2 to 5.4) with the harness's 16-token answer, 12 with thinking on
    strict
    the general door's strict is the fc-door file's rule: the answer names one candidate
    reader
    claude haiku 5.5
    cost
    $0.25 to file 18,332 facts once, about $0.0002 a question
    files
    packages/bench/results-mab-fc-haiku-5-5.json; the general door, packages/bench/results-mab-fc-door-haiku-5-5.json
    • phases is a read path built for this test; the general door is the one to set beside the benchmark's own table
    • at least 10 of the 100 gold answers exist in the pool only as a value a later fact replaced, so the ceiling is at most 90
    • version 1.1 fixed two bugs the held-out run exposed and re-ran every cell; version 1 scored 61

  • learningbelow long context

    learning from examples

    memoryagentbench test-time learning: five classification sets (banking77, clinc150, nlu, trec coarse and fine), thousands of labelled examples, 100 questions each.

    81.695% interval 78.0 to 84.8
    • the leaderboard door · claude haiku 5.581.6
    • best in the paper's table (claude-3.7-sonnet, long context)89.4
    • gpt-5-mini, long context84.0
    • best memory agent in the table (qwen3-embedding rag)78.0
    • bm25 in the table, gpt-4o-mini reading75.4
    • mem0 in the table32.4

    7.8 under the published line it is measured against.

    the same reader, run by us claude haiku 5.5 in front of each, so only the memory moves.

    • 0.0no memory, the question alone
    • 79.4bm25, the harness's own, 512-token chunks, top 1011,050 tokens a read
    • 83.2the same bm25, 4,096-token chunks, as the paper set it here38,682 tokens a read
    • 70.6a small embedder (bge-small) on the same chunks, top 109,078 tokens a read
    • 82.8the door held to bm25's token budgetits whole top 100 is under bm25's budget, so this row is not budget-matched1,994 tokens a read
    • 81.0the door without its query expansion1,996 tokens a read

    with the same reader, the harness's bm25 is level with the door: 79.4 on 512-token chunks and 83.2 on 4,096 as the paper set it, against 81.6 on the same 500 (paired, p 0.31 and 0.48), reading more than five and nearly twenty times the tokens. it is level in sum only: by set, banking77 goes to the door and trec fine to bm25 on 4,096-token chunks. with no examples the reader scores 0, since the labels are numbers it cannot guess.

    • banking7796 of 100
    • clinc15093 of 100
    • nlu84 of 100
    • trec coarse76 of 100
    • trec fine59 of 100
    data
    ai-hyz/MemoryAgentBench, the five icl sets
    published
    MemoryAgentBench v4 (arXiv 2507.05257v4), table 3 (MCC)
    run
    2026-10-22, piloted on 20 of two sets, then all 500 once
    scoring
    exact match after reading "label: 43" as 43, as the harness's readme recommends
    reader
    claude haiku 5.5
    read
    2.0k tokens per answer
    same reader
    2026-10-25, all 500 per arm
    files
    packages/bench/results-mab-door-haiku-5-5.json; the same-reader rows, packages/bench/results-mab-controls-haiku-5-5.json
    • under the paper's strict parse every answer written as the prompt asks scores 0
    • by set, paired: on banking77 the door is right on 15 that bm25 on 512-token chunks misses and misses 3 it gets (p 0.0075); on trec fine bm25 on 4,096-token chunks is right on 29 the door misses, against 9 (p 0.0017)
    • above every memory system in the table and below most long-context rows (gpt-4.1-mini's 75.6 is under it), with a 2026 reader against 2025 ones; with the same reader, plain bm25 is level with it

  • learninglevel with no memory

    films to recommend, from a long history

    memoryagentbench Recom, the second half of test-time learning: a long run of film-recommendation dialogues is read first, then for each of 200 new asks the reader lists films, scored by recall at 5.

    15.795% interval 11.1 to 20.5
    • the leaderboard door · claude haiku 5.515.7
    • best in the paper's table (claude-3.7-sonnet, long context)18.3
    • best memory agent in the table (text-embed-3-large rag)16.2
    • bm25 in the table, gpt-4o-mini reading13.6
    • gpt-5-mini, long context13.2
    • mem0 in the table10.0

    2.6 under the published line it is measured against.

    the same reader, run by us claude haiku 5.5 in front of each, so only the memory moves.

    • 14.4no memory, the ask alone
    • 14.3bm25, the harness's own, 4,096-token chunks, top 10

    no measurable memory effect: with no memory at all the same reader names films as well, 14.4, and the harness's bm25 14.3; the door against no memory is a paired 1.3 (interval −3.1 to 5.6). the 15.7 is mostly the reader.

    data
    ai-hyz/MemoryAgentBench (MIT), the recsys set
    harness
    HUST-AI-HYZ/MemoryAgentBench at 5380260: its prompt, its reader layout, its recall at 5 (utils/eval_other_utils.py)
    published
    MemoryAgentBench v4 (arXiv 2507.05257v4), table 3 (Recom, Recall@5)
    run
    2026-10-25, piloted small, then all 200 once
    reader
    claude haiku 5.5 (the paper's memory agents ran on gpt-4o-mini)
    read
    20.5k tokens a read for the door, the prompt included; 59.8k for bm25's top 10
    recall
    the door at 1: 7.1, at 10: 23.8
    file
    packages/bench/results-mab-lrttl-haiku-5-5.json
    • the films the reader writes are matched to the set's list by edit distance, as the harness does; a tie, which the harness breaks at random, goes to the exact name
    • the harness's answer length is in gpt-4o-mini tokens; haiku's tokenizer counts this set's text at 1.49 to 1.66 times as many, so the length is 1.6 times the harness's
    • against bm25 the door is +1.4 (interval −3.2 to 5.8): every memory here, ours included, is level with the reader alone

  • understandingbelow plain bm25

    a whole book, summarised

    memoryagentbench ∞Bench-Sum, from long-range understanding: summarise a novel that went in a passage at a time, scored by key points against an expert's summary.

    24.795% interval 20.7 to 28.5on the 49 books bm25 is scored on: 26.4
    • the leaderboard door · claude haiku 5.524.7
    • best in the paper's table (gpt-5-mini, long context)56.3
    • claude-3.7-sonnet, long context52.5
    • best memory agent in the table (bm25)19.0
    • mem0 in the table4.8

    31.6 under the published line it is measured against.

    the same reader, run by us claude haiku 5.5 in front of each, so only the memory moves.

    • 33.0bm25, the harness's own, 4,096-token chunks, top 10on 49 of the first 50 books: one read was refused twice and is left out

    with the same reader and judge, the harness's bm25 summarises better: 33.0 against the door's 26.4 on the same 49 books, a paired difference of −6.5 (interval −12.9 to −0.2), just clear of zero, and bm25 reads 1.7 times the tokens. the door answers a summary ask with a search, as it answers everything.

    data
    ai-hyz/MemoryAgentBench (MIT), the ∞Bench-Sum set
    published
    MemoryAgentBench v4 (arXiv 2507.05257v4), table 3 (Summ, F1)
    scoring
    the paper's f1 is a judged key-point f1: fluency, times the balance of key points recalled and sentences that hold, against the expert's summary
    run
    2026-10-25: the door on all 100 books, bm25 on the first 50, since the judge costs about $0.06 a summary; one bm25 read was refused by the api, asked once more, refused again, and left out of the score
    reader
    claude haiku 5.5
    judge
    claude sonnet 5.5 with the harness's three judge prompts, byte for byte (the harness used gpt-4o)
    parts
    the door: fluency 99, key points recalled 25.2, sentences that hold 28.1
    read
    36.1k tokens a read for the door, the prompt included; 62.8k for bm25's top 10
    file
    packages/bench/results-mab-lrttl-haiku-5-5.json
    • the summary ask goes through search as it is and pulls passages from all over the book; that is the honest number for the door
    • the door and bm25 read different numbers of books; set them side by side only on the 49 both are scored on
    • with the refused read scored 0, as it first was, bm25 reads 32.3 and the paired difference is −5.9 (interval −12.1 to 0.3), so how far past noise this gap sits turns on one book
    • judged by Claude, not by gpt-4o

  • understandingbelow plain bm25, not significant

    a detective novel, solved

    memoryagentbench DetectiveQA, from long-range understanding: 71 questions on ten detective novels, four options each, whose clues are spread through the book.

    71.895% interval 60.5 to 81.0
    • the leaderboard door · claude haiku 5.571.8
    • best in the paper's table (gpt-4o, long context)77.5
    • gpt-5-mini, long context76.1
    • best memory agent in the table (mirix on gpt-4.1-mini)62.0
    • bm25 in the table, gpt-4o-mini reading52.1
    • mem0 in the table36.6

    5.7 under the published line it is measured against.

    the same reader, run by us claude haiku 5.5 in front of each, so only the memory moves.

    • 2.8no memory, the question aloneit named no option on 69 of the 71, so it sits under chance (25)
    • 80.3bm25, the harness's own, 4,096-token chunks, top 10

    with the same reader, the harness's bm25 solves more as a point estimate: 80.3 against 71.8 on the same 71 questions, a paired difference of −8.5 (interval −16.9 to 0). only 10 questions split them, 2 for the door and 8 for bm25 (exact p 0.109), so it is not significant, and bm25 reads 2.8 times the tokens.

    data
    ai-hyz/MemoryAgentBench (MIT), the DetectiveQA set
    published
    MemoryAgentBench v4 (arXiv 2507.05257v4), table 3 (DetQA)
    scoring
    accuracy over the four options, the letter in the answer field
    run
    2026-10-25, piloted small, then all 71 once
    reader
    claude haiku 5.5
    read
    21.3k tokens a read for the door, the prompt included; 60.3k for bm25's top 10
    file
    packages/bench/results-mab-lrttl-haiku-5-5.json
    • the questions ask for a json answer, which the readme's exact match can never equal: by it every arm scores 0, so the score is the chosen option; a looser match gives the door 67.6 and bm25 74.6
    • 71 questions on ten books, so the interval is wide

  • overallthe paper's overallunder the published line

    memoryagentbench, all four kinds

    the paper's overall: accurate retrieval, test-time learning, long-range understanding and selective forgetting, each averaged, then the four averaged, each kind weighed the same.

    54.0the general door with a thinking reader on fact consolidation: 58.7, strict 58.5only with phases, the read path built for fact consolidation: 65.1
    • the leaderboard door · claude haiku 5.554.0
    • best in the paper's table (gpt-5-mini, long context)60.6
    • claude-3.7-sonnet, long context49.6
    • best memory agent in the table (hipporag-v2)41.6
    • bm25 in the table, gpt-4o-mini reading41.5
    • mem0 in the table21.1

    6.6 under the published line it is measured against.

    the same reader, run by us claude haiku 5.5 in front of each, so only the memory moves.

    • 52.5plain bm25 as the paper set it, on every testfact consolidation through bm25 over the top 10 fact lines; summaries on 49 books (52.5 too with the refused one scored 0)

    with the same reader, plain bm25 as the paper set it adds up to 52.5 the same way, where the paper's bm25 row, read by gpt-4o-mini, is 41.5: most of the distance from the paper's memory agents is the reader, and the door is 1.5 over plain bm25 on the overall. bm25 here reads fact consolidation as its top 10 fact lines, not as the harness's own agent.

    how
    the mean of the four kind averages, each kind rounded to one decimal first, as the paper's own 60.6 needs: gpt-5-mini's 74.4, 48.6, 66.2 and 53.0 average 60.55, which rounds to its 60.6 (unrounded, 60.5)
    ours
    the general door on every test, the same quick reader throughout: accurate retrieval 85.5, test-time learning 48.6, long-range understanding 48.3, selective forgetting 33.5, each from exact values
    beside it
    the general door with thinking on for fact consolidation alone (52.5 there) gives 58.7, or 58.5 by fact consolidation's strict rule; phases on fact consolidation (78.0 there), a read path built for that one test, gives 65.1
    parses
    detectiveqa is scored by the option chosen and test-time learning's mcc with "label:" read off; the harness's own exact match is 0 for every arm on both, so every overall here rests on those two readings
    same reader
    plain bm25 added up the paper's way from this page's same-reader cells: accurate retrieval 66.3 (512-token chunks on the three synthetic sets, 4,096 on the novels), test-time learning 48.7 (mcc 83.2, films 14.25), long-range understanding 56.6 (summaries 33.0 on 49 books, detectiveqa 80.3), selective forgetting 38.5 (74 and 3); 52.5
    published
    MemoryAgentBench v4 (arXiv 2507.05257v4), table 3, the overall column
    reader
    claude haiku 5.5 on every test
    files
    packages/bench/results-mab-lrttl-haiku-5-5.json (its overall block), from results-mab-door-haiku-5-5.json, results-mab-fc-door-haiku-5-5.json and results-mab-fc-haiku-5-5.json
    • the general door's 54.0 is the number to set beside the paper's table; 65.1 puts a read path built for one test into an average of all of them, so nothing on this page leads with it
    • every kind weighs the same: selective forgetting through the general door (33.5) pulls it down most; recommendation shows no measurable memory effect, and on long-range understanding the door is below plain bm25 with the same reader: on summaries just clear of noise, on detectiveqa within it
    • the results file's own overall block prints 58.8 for the thinking reader, from a recommendation score rounded before it was averaged; from exact values it is 58.7
    • the reader is a 2026 model; the paper's memory agents ran on gpt-4o-mini and its long-context rows are 2025 models
    • the same-reader bm25 is approximate: its fact consolidation is bm25 over fact lines rather than the harness's agent, and its summaries miss one book

  • the persona pilot of 60; its interval spans both lines

    personamem-v2, implicit preferences

    multiple choice: which answer fits a person whose preferences were only ever implied, over a 32k-token history.

    48.3%95% interval 36.2 to 60.7
    • the leaderboard door · claude haiku 5.548.3
    • the paper's own agentic memory55.0
    • frontier models with the whole history, up to48.0

    6.7 under the published line it is measured against.

    data
    bowen-upenn/PersonaMem-v2 (CC-BY-4.0), the text benchmark
    run
    2026-10-22, 60 queries in a seeded sample of the 5,000
    memory
    the leaderboard door, the same as every test above, not netti-1
    reader
    claude haiku 5.5 with the official multiple-choice instruction
    file
    packages/bench/results-personamem2-door-haiku-5-5-n60.json
    • a pilot of 60; not run further because it is not yet a number worth running in full
    • its interval, 36.2 to 60.7, spans both published lines, so it says nothing either way about them
    • this is the person model's job, and netti-1 has not been run on it

the person model

and it is learning people.

40 synthetic people, 90 days of their lives across Claude, ChatGPT, Gemini, Cursor, email and agent logs, and about 140 questions each, scored against what each person would really choose.

netti-1 is the person model, netti-0 in the lab; the product does not run it today. the ladder sets it beside mem0 open source alone, in an exploratory comparison; the headline counts it against all four vendors' engines run on likeness, with Holm's correction over the 36 comparisons that leave out what they will need, which likeness-1.1's counting keeps out of every count. of the five families it holds on against all four, netti-1 also beats that family's floor, a simple statistic of the person's own past, on one: what they mean.

netti-1mem0 open source 3.3.1, its own code with our model slot, read by the same readernats gained over no memory, higher is better

  • what they choose+0.25 +0.15
  • what they do next+0.35 +0.23
  • what they mean+0.53 +0.08
  • what changed+0.81 +0.42
  • when it changed+0.70 +0.44
  • whose words+0.34 +0.14
  • what they will need+0.07 −0.13
  • restraint+0.19 +0.11
  • what weighs more+0.08 +0.19
  • no false insight−0.39 −0.39

after a forget, the forgotten words were served again to

of 40 people. mem0 open source also still stored them for 34. tokens a read: netti-1 657, mem0 open source 1,288.

bench
likeness-1, pre-registered; this comparison was run after the test split was revealed, so it is exploratory
people
40 synthetic test people, scored once
netti-1
netti-0 version 0.3 in the lab, shown here by its outward name
mem0
mem0ai 3.3.1 open source: its own extraction, store and search; claude haiku 5.5 in its model slot, its own local embedder
reader
claude haiku 5.5 for both
paired
per person, bootstrapped, each comparison's own 95% interval: netti-1 ahead on 8 of 10; mem0 ahead on what weighs more; no false insight even. with Holm's correction over the 36 comparisons with the four engines, what they will need left out: netti-1 ahead on 6 (restraint drops), mem0's lead on what weighs more holds; of those 6, netti-1 also beats the family's floor on 2, what they mean and whose words
the other engines
letta code 0.34.24 and graphiti 0.31.2, zep's open-source engine, both open source, and supermemory's self-hosted server 0.0.9, not open source, each its own code with claude haiku 5.5 in its model slot, a local embedder and our read: netti-1 ahead on 8, 9 and 8 of 10 with intervals clear of zero, the same 8 against all four; letta and supermemory ahead on what weighs more, graphiti on no false insight. with Holm's correction: 6, 8 and 6, the same 5 against all four; letta's and supermemory's leads on what weighs more do not hold, graphiti's on no false insight does. of those 5, netti-1 also beats the family's floor only on what they mean
the reads
netti-1 reads with the answer options and makes a read-time model call on some decision reads; the engines search with the ask alone. letta's search reads quoted words as a phrase, so its recall was empty on asks that quote a sentence; read without the quotes, letta was nearer on what they mean and restraint, and still behind netti-1 on both
mem0's store
still stored for 34 of 40: 17 in its current memories, 17 more only in its change log (history.db)
files
docs/research/person-models/field/v6-paired.json, v6-noquote.json, v6-forget.json, v6-economy.json; the Holm counts and the floor tests from packages/bench/person/out/analyze11/v6-holm-1.1-counts.json
  • our own benchmark, on synthetic people
  • mem0 ran with Haiku in its model slot, not its default gpt-5-mini
  • every memory, ours included, does worse than no memory on false insight
  • graphiti is not zep's cloud product, and supermemory's self-hosted server is not its hosted one; their numbers are on likeness's own page with each engine's caveats
  • likeness's own plain retrieval arm reads what weighs more and no false insight better than netti-1, both intervals clear of zero; it and the other memories run on likeness are on that benchmark's own page

here netti-1 is set beside mem0 open source alone. how the benchmark works, why it asks what recall cannot, and every memory it sets side by side, letta, zep and supermemory among them, are on its own page: likeness.

how these were run

the system, and the rules.

innernet memory, the leaderboard door. every conversation chunk is added whole (the raw messages are kept) and read once by Claude Haiku 5.5, which writes the facts worth keeping with absolute dates. a search ranks the raw messages and the facts together: BM25 over the question and one Haiku query expansion, fused by reciprocal rank, top 100. no embeddings, no reranker. it is a research door built for benches like these, not innernet as shipped; where the shipped path has been run on the same questions, its row says so.

  • published, and same reader

    a published row is the number another paper or table prints, named as such, with its reader; most are other, older readers, so a gap to one is never a ranking. a same-reader row is one we ran: no memory, plain retrieval, a small embedder, or our door with one part changed, in front of the same reader, so only the memory moves, except where a cell says it changes how the reader reads. on likeness we ran the vendors' own open-source or self-hosted engines, each with our model slot, embedder and read.

  • the leaderboard door, and innernet as shipped

    most rows read the leaderboard door, a research memory built for benches like these, and say so beside the score. where innernet as shipped has run the same questions, written and read the way a connected ai reads it, its number sits on that row beside the door's, and the two are not the same.

  • their prompts, their scorers

    each benchmark is run with its own prompts and graders, copied from its harness, and any step that is ours is written down.

  • piloted, then run once

    a small pilot finds the plumbing bugs. then the full set runs one time, with nothing tuned on it, except where a row says so: the door was tuned on 150 longmemeval-s questions, and phases on the smallest fact consolidation cells.

  • the reader named

    every number says which model read and which model judged. a different reader is a different number.

  • running says running

    a test still being run shows no number, never a zero. a number lands from its results file once the full set has run.

the bench's own ledger (packages/bench/lab-cache/usage-ledger.jsonl) holds $37.70 of Claude for 21 and 22 oct, every run on those days, a few of them not on this page (pasb, laundering, a rerank and an answer-step try), and $46.79 for 25 oct (utc): the same-reader rows, the four new columns and the shipped path at its three builds, pilots and retries included, the $2.46 of what-if diagnostics left out. MemoryBench's own answering and judging, about $3, and the likeness runs sit outside it. source: packages/bench/BENCHMARKS.md, sections 000 to 000d (runs of 2026-10-21 and 2026-10-22), the runs of 2026-10-25 (results-mab-controls, results-mab-lrttl, results-mab-fc-door, results-mab-today-lme, results-mab-today-lme-main and results-mab-today-lme-1e9b3606), the results files named on each test, MemoryAgentBench v4 (arXiv 2507.05257v4, 28 jun 2026, table 3) for every published row of that benchmark, and docs/research/person-models/field (its v6 files) for likeness, with its Holm counts and floor tests from packages/bench/person/out/analyze11/v6-holm-1.1-counts.json. every average is taken from exact values and rounded half-up to one decimal (the overall averages the four kinds after rounding each, as the paper's 60.6 requires). read with claude-haiku-5-5; the memory is packages/bench/aml/memory.mjs.