Role-Play Bench: A Transparent Independent Reproduction and Exploratory Validation Report

1 Introduction

Role-play evaluation lacks a single correct answer: many responses can be appropriate for a character, but responses that contradict the character, the world or the dialogue history can be identified with comparatively high agreement. MiniMax built Role-Play Bench on this observation, describing misalignment as "surprisingly objective" while alignment is subjective [1]. The benchmark evaluates long, multi-turn trajectories because many failures, such as repetition, forgetting and drifting length, only appear after the first 20 turns [1, 2].

This document has four purposes:

1. to describe the original benchmark strictly as specified by its authors (Section 2);
2. to describe our implementation, separating what follows the sources, what we reconstructed, and what we changed (Section 3);
3. to report exploratory checks of our implementation, including negative and inconclusive results (Section 4);
4. to list deviations, limitations and the configuration needed to reproduce our numbers (Sections 5–7 and the appendices).

Throughout, original refers to MiniMax's method as published, implementation to our reproduction of it, and change to a departure we introduced deliberately. The report does not claim that the implementation is more accurate than the original, whose judge is unpublished; where the two disagree, it reports what reading the dialogues showed, case by case.

1.1 How the implementation was built

Reproducing a benchmark whose judge and scoring rule are unpublished requires decisions that the sources do not dictate. We made them by a fixed procedure, so that each component can be traced to a source, to evidence in the released data, or to a stated assumption:

1. Source audit. Every statement about the method was extracted from the three primary sources and classified as specified, inferable from the released data, or unspecified (Sections 2.1–2.6).
2. Reconstruction from released data. Where the release constrains a component, the reconstruction follows it: the pass/fail rule for Basics and Logic follows the granularity of the released scores (Section 2.5), and the simulated user's turn lengths follow the user side of the released dialogues (Section 3.3). Components the release does not constrain are documented assumptions (Section 5).
3. Calibration on held-out scenarios. Our judges score MiniMax's released dialogues, so their verdicts can be compared with MiniMax's on identical text. Any parameter fitted to those verdicts is fitted on four scenarios and evaluated on six others (Sections 4.1, 4.8).
4. Controlled experiments. Generation conditions (user side, serving host, temperature, reasoning, output limit) are varied one at a time on identical scenarios and compared with paired bootstrap intervals [22] (Sections 4.2–4.5).
5. Reading the disagreements. Where our verdicts and MiniMax's diverge, dialogues are read and each disputed failure is classified as genuine or spurious before any rule is changed (Sections 4.1, 4.8).
6. Versioning and disclosure. The result is frozen as a versioned protocol; each run stores a snapshot of it, and the full protocol, including every prompt, is public (Section 7, Appendices A–B).

1.2 What the evidence supports

Following argument-based validation [19, 20], we state the inferences that lead from a dialogue to a model's position under this implementation and the evidence for each (Figure 2, Section 4). In brief, and on exploratory evidence: recorded failures were genuine in the samples we read; our judges' model-level verdicts on the released dialogues resemble the original judges', with differences in Basics and Logic that are only partly explained; one rerun moved a score by less than a point; and widely separated models are distinguished. Not yet supported: that small differences between the highest-scoring models under our generation protocol are meaningful, that scores reflect the experience of human users, and that the simulated-user protocol reproduces the original orderings.

1.3 What is disclosed

Every step from a dialogue to a score is open to inspection. Table 1 compares what the original release and this implementation disclose; disclosure makes a score checkable, not correct. The main properties are these.

1. Disclosed judges. The official panel consists of two reasoning models from different developers, two reasoning models, with pinned model identifiers. Each scores every segment independently, the scores are averaged, and disagreements beyond fixed thresholds are flagged, not resolved. Both developers also have models among those evaluated, and LLM judges can prefer their own family's outputs [21, 7]; an exploratory check found no significant family effect but cannot exclude one of several points (Section 6). The screening and official panels order models similarly on the same 44 dialogues (ρ = 0.94; Section 4.1).
2. Judges detect; a published rule scores. Judges never output a score. They list failures, each with its round, a verbatim quotation and a reason, and a deterministic, published rule turns the list into scores (Section 3.6). Holistic ratings by LLM judges agree poorly with humans in role-play and tend to exceed human ratings [5, 12, 9]; checklist-style detection agrees better [12]. Every deduction in a run on this platform is stored with the quotation the judge gave; quotations are not yet checked automatically against the dialogue.
3. The whole dialogue is in view. Judges read the complete preceding dialogue with each segment, so forgetting and contradictions across 100 turns are detectable, not only errors within a 20-turn window.
4. Controlled generation conditions. Every model meets a simulated user with the same persona and the same sequence of turn-length budgets per scenario and sample. Since 26 October 2026 the simulator model is also fixed per scenario and sample, so every model meets the same simulator in the same slot (Section 3.3); in earlier runs it was assigned per session, so two models could meet different simulator models on the same scenario (3–6 of 10 scenarios per pair in Section 4.5). Each model is served by one pinned host with a sufficient output limit, and its reasoning setting is labelled. In self-play experiments, the serving host moved one model's score by 5.6 points and the sampling temperature by up to 11.0; because the evaluated model also plays the user in self-play, these are effects on the whole generation pipeline (Section 4.3).
5. Reported uncertainty and corrections. Every score carries a bootstrap confidence interval; small samples are labelled and kept off the leaderboard; checks are reported with their sample sizes, including negative results and corrections (Section 4). Where our judges and the original disagree, we read the dialogues: on the three models ranked lowest on MiniMax's leaderboard, 13 of 14 sampled failures that only our judges recorded were genuine (Section 4.1).

These properties make the scores checkable; they do not by themselves make every score correct. The checks have also found open issues, reported alongside the strengths: a classification of viewpoint switches that differs from the original's, addressed by a candidate revision not yet adopted (Section 4.8), and differences in Basics verdicts that have not yet been examined (Section 6).

2 The original benchmark

2.1 Sources

Three primary sources describe Role-Play Bench: the MiniMax-M2-her technical blog post [1], the dataset card and files on Hugging Face [3], and Appendix A with Tables 7–9 of the HER paper [2], a joint work of Fudan University and MiniMax that follows the benchmark's "official protocol" as an external evaluation. The HER paper does not describe the judge or the scoring procedure; its evaluation details (judge model, decoding settings, a severity-based deduction formula and human-agreement figures) refer to the CoSER benchmark, not to Role-Play Bench, and we do not attribute them to it.

2.2 Design

• Negative evaluation. The benchmark detects failures ("misalignment") instead of defining ideal answers [1, 3].
• Scenarios. 100 character (NPC) settings per language were sampled uniformly from a structured taxonomy built over an internal library of more than one million NPC and user prompts, and the data team wrote a scenario for each [1]. The HER paper refers to "an internal dataset of 100 dialogue prompts" [2]. The public release is described as a partial subset [3].
• Dialogue generation. Model-on-model self-play: the evaluated model plays both the character and the user; external models received user prompts refined on real online user behaviour. Each scenario is run for 100 turns, three times [1, 3]. The authors report that a fixed user model produced results statistically similar to self-play, without giving the test or the numbers [1].
• Judging. Only the character side is scored. Dialogues are split into 20-turn chunks; the scoring stack includes consistency checks across samples and manual calibration against human preference data [1, 3]. The judge model, its prompts and calibration statistics are not published.

2.3 Failure taxonomy

Table 2 lists the sub-dimensions as defined in Table 9 of [2], grouped as in its Table 7. The blog post and the dataset card additionally describe a Knowledge dimension (adherence to the world's physical or magical rules) and formatting glitches under Basics [1, 3]; neither appears as a separate sub-dimension in Table 9. The released score files contain six dimensions: basics, logic, knowledge, diversity, content logic and interaction [3].

The assignment of the five Worlds sub-dimensions to Basics and Logic in Table 2 follows the prose of [2] (Appendix A) and of the blog post; the paper itself does not state it.

2.4 Scoring as published

The overall score is 0.5 × Worlds + 0.25 × Stories + 0.25 × Preferences (Eq. 3 of [2]). Stories and Preferences are the means of their sub-dimensions (caption of Table 7 of [2]). The sources do not specify how detected failures become scores, how the Worlds score is formed, how scores are aggregated over chunks, runs, scenarios and languages, or how the reported 95% confidence intervals are computed.

2.5 Released data

The release (revision 3c1be2a5) contains, per language, the scenario seeds, the dialogues and session-level scores: 50 Chinese and 45 English scenarios, 11 models, three runs per scenario, 3,135 dialogues of 100 turns, and one 0–100 score per dimension per session [3]. The card states that the dialogues were synthetically constructed by the data team, intentionally skewed toward chat-oriented role-play, and that user-side fields were varied in parts of the data. The licence is stated inconsistently: the metadata header says Apache-2.0, while the card's licence section says CC BY 4.0 / CC BY-SA 4.0.

Two properties of the released scores matter for reproduction. First, about 97% of the basics and logic session scores are multiples of 20, which is consistent with a pass/fail decision per 20-turn chunk over five chunks; the other dimensions take graded values. This is our inference from the data, not a documented rule. Second, the published leaderboard cannot be recomputed from the released scores: with the published group weights, the released sessions give MiniMax-M2-her about 87 instead of the published 84.65, and the order of two Claude variants reverses (Table 3). The published values were presumably computed on the full, partly unreleased scenario set or with an unpublished aggregation.

2.6 Unspecified components

The following are required for an exact reproduction and are not specified in the sources:

1. the 5 unreleased scenarios per language and how the released ones were selected;
2. the user-simulation prompts, persona handling, length control and decoding settings, and which model plays the user for external models;
3. the decoding settings of evaluated models (temperature, maximum output length, reasoning budget);
4. the judge model(s), prompts, decoding settings and the context given with each chunk;
5. the mapping from failures to scores (per chunk or per turn; counts, pass/fail or severity) and the formation of the Worlds score;
6. aggregation over chunks, runs, scenarios and languages, and the confidence-interval method;
7. the human-calibration data, procedure and agreement statistics.

2.7 Related work and its bearing on this implementation

Role-play benchmarks. Earlier benchmarks score single responses or short, pre-written contexts against character profiles [4, 5, 6]. Multi-turn and simulation-based designs followed: RMTBench uses pre-generated user turns over long dialogues [8], and CoSER simulates multi-character scenes and has critics count flaws with severity penalties [7]. Role-Play Bench differs by generating 100-turn interactive trajectories and judging only the character side for predefined failures; a survey places these approaches in context [23].

LLM judges. Prompted general-purpose judges correlate only moderately with human ratings of role-play: in CharacterEval, GPT-4 reached Pearson r ≈ 0.36–0.39 against 0.63 for a reward model trained on human labels [5]; in HumanLLM, holistic judgements correlated at r = 0.43–0.61 while checklist judgements reached 0.90 [12]. Judges tend to rate agents higher than humans do, particularly agents trained for social interaction and late in long interactions [9, 10, 11], and can prefer outputs of their own model family [7]. Fine-grained detection of out-of-character content finds failures that response-level scores miss [13]. These results support failure-based (checklist-like) judging, which is what Role-Play Bench uses, and they motivate our comparison with the original judges' verdicts (Section 4.1) and our reporting of a self-preference risk (Section 6).

User simulation. Default LLM user simulators are reported to be overly cooperative and consistent; replacing the simulator can reorder model rankings, and simulators can be validated against real-user traces with discriminators or distributional measures [14, 18]. Simulated personas drift over long dialogues [15], instruction-tuned models are argued to be structurally poor simulators of people [16], and the composition of a persona pool biases outcomes [17]. Our simulator (Section 3.3) addresses length and coupling, which we can measure from the released dialogues, but it has not been validated against real users in the sense of these works; we list this as a limitation and as planned work.

3 Our implementation (protocol v6.0)

The evaluation standard is fixed per benchmark and versioned; every run stores a snapshot of the protocol, prompts, judge configuration and prices in force when it started, so scores remain attributable after later changes. Appendix B lists the versions.

3.1 Data and run configurations

We use the released seeds at revision 3c1be2a5 unchanged. A full-protocol run covers all 95 scenarios (50 Chinese, 45 English) × 3 independent samples × 100 turns = 285 sessions, chunked into 20-turn segments. Two smaller configurations exist for development: a quick sample (5 scenarios per language × 1 sample × 40 turns) and a standard sample (15 × 1 × 100). They are labelled as screening results with their confidence interval and are never eligible for the leaderboard (Section 4.4 shows why).

3.2 Dialogue generation

Each session starts from the seed's prologue (character turn, round 0) and initial user input (round 1); the character speaks at even rounds. The evaluated model receives a system prompt rendered from the seed (character and user settings; Appendix A) and the dialogue history; it receives no instruction about response length. Length is treated as part of what is measured: the only limit is the maximum output below, so a model aligned to answer briefly like a person in chat is neither corrected nor penalised for brevity, and a verbose one is not rewarded for length; the judges count failures, not words. Requests set a maximum output of at least 8192 tokens because reasoning tokens count toward the limit and a 1,024-token limit truncated 61% of one model's replies in an earlier version; the finish reason of every reply is stored and truncation rates are reported. Reasoning ("thinking") settings are passed explicitly and the run name carries them, so the same model with reasoning on and off is ranked separately. A reply blocked by the provider's content moderation is recorded as no response and judged as AI silence.

3.3 User simulator (change)

The original uses self-play. We use a fixed user simulator (two models, reasoning effort provider default), for two reasons we measured (Section 4.2–4.3): in self-play the user side inherits the evaluated model's verbosity, so user turns grow with the model's replies (one model reached 2,401 words per user turn); and the score then mixes the model's quality as a character with its behaviour as a user.

The simulator plays a persona fixed per scenario and sample. Its length per turn is drawn from a distribution of real user messages, independently of the character's length; the reference is the user side of MiniMax-M2-her's released dialogues, which the blog describes as reproducing real user habits [1] (Chinese median 9 characters, English median 5 words). A share of turns are action-only. A reply longer than twice its budget is rewritten once. Self-play and the earlier simulator remain available as experimental controls; such runs are never eligible for the leaderboard.

Which simulator model plays the user (26 October 2026). The simulator model is fixed by scenario and sample: a language's scenarios in id order, three samples each, alternate between the two models. Every model under test therefore meets the same simulator in the same slot, each simulator plays half of each language (75 and 75 Chinese sessions, 68 and 67 English), and every scenario meets both. Before this, a hash of the session id chose the model; in a full run of 285 sessions that gave one simulator 85 of the 150 Chinese sessions, and the assignment differed between models on the same scenario, which adds noise to comparisons between models. Runs created earlier keep their assignment, so resumed sessions keep their simulator.

Moderation on the user side. When the provider's content moderation blocks the simulated user (in a full run of 285 sessions, 2 of 14,250 user turns, both in one session), the same model cannot produce the turn on retry. That one turn is written by the other simulator model, the switch is logged and the message records which model wrote it; the next turn returns to the session's simulator. The rule is the same for every model under test and changes nothing on the side of the evaluated model or the judges.

3.4 Generation conditions (change)

Models served through a router can be routed to different hosts with different quantization and default sampling settings within one run. In self-play, pinning the host shifted a model's score by 5.6 points (Section 4.3). A run is therefore eligible for the leaderboard only if the evaluated model is pinned to a single host with fallbacks disabled and the stored replies confirm a single host; direct provider endpoints satisfy this by construction. Temperature is the endpoint's configured value (provider default unless set) and is recorded per run.

3.5 Judges (reconstruction)

Judges read the scenario setting, the whole preceding dialogue (undefined turns) and one 20-turn segment, and return a list of failures with round, type, quoted evidence and reason. Two panels exist (Table 4). The official panel scores full-protocol runs; the screening panel scores development runs at lower cost and splits dimensions between a reasoning model and a decision model that answers one yes/no question per character turn and failure type, compared against a threshold.

When two judges score the same dimension, their scores are averaged with the stated weights. A session is flagged as disputed when the judges differ by at least 15 points on a dimension (for pass/fail dimensions, when they disagree on at least two segments), and a segment when they differ by at least 30; disputes are reported, not resolved.

3.6 Scoring and aggregation (reconstruction)

1. Seventeen failure types are detected: the fifteen of Table 2, a world-rule violation for Knowledge, and formatting glitches under Basics (both described in [1] and present as released score columns).
2. Per segment, each sub-dimension scores 100 − 10 × (number of character turns showing the failure), floored at 0. Basics and Logic are pass/fail per segment (0 if any of their sub-dimensions occurs, otherwise 100), following the inference in Section 2.5. Knowledge is its single sub-dimension; Diversity, Content logic and Interaction are means of their sub-dimensions.
3. A session's dimension score is the mean over its segments.
4. Worlds = mean of Basics, Logic and Knowledge (our assumption; unspecified in the sources). Stories = (4 × Diversity + 2 × Content logic) / 6, which equals the mean of its six sub-dimensions as in [2]. Preferences = Interaction. Overall = 0.5 Worlds + 0.25 Stories + 0.25 Preferences.
5. A model's score is the mean over sessions, Chinese and English pooled. The 95% confidence interval is a percentile bootstrap over sessions (1,000 resamples; reported from 5 sessions upward). Because the three samples of a scenario are correlated, this interval is somewhat too narrow; on the released data, clustering by scenario raises the standard error by 14–34% for the three models checked.

3.7 Leaderboard eligibility

A run enters the leaderboard only if it uses the full protocol, the current protocol version, has every session scored, and was generated by a single host (Section 3.4). The leaderboard currently lists only the 11 models of the public release, with scores recomputed from the released session scores (Table 3); no run generated on this platform has yet been published to it.

3.8 Supplementary experience-quality score (change, not part of the overall)

Because the failure-based score saturates for models that make few failures (Section 4.5), runs created since 2026-10-26 also receive an experience-quality score. One judge (a reasoning model) rates each 20-turn segment from 1 to 10 on five anchored scales (characterization, expression, narrative, responsiveness, naturalness); the mean is rescaled to 0–100. It is reported beside the overall score and does not affect the overall or the ranking. Two different judges agreed at Pearson r = 0.86 on 320 segments; agreement with human judgement has not been established.

3.9 Session rule: replies in the wrong language (change, 26 October 2026)

A session in which the evaluated model mostly answers in the other language of the scenario scores 0 on every dimension. A reply counts as in the wrong language when more than half of its letters belong to the other script (CJK characters against Latin letters), so names, loanwords and short quotations do not count; the session scores 0 when at least half of its replies from round 2 on do. The judges' own scores are kept beside the 0 and shown on the session page. The rule is part of this benchmark's standard and applies to every model, including the 11 models of the public release, whose stored results were re-scored with it.

Why. In a full run of Doubao-Seed-Character, three English sessions were answered in Chinese from the first reply on, and the simulated user followed into Chinese from its first turn; the dialogue the judges saw was a Chinese role-play, and the failure to keep the scenario's language would otherwise cost only part of the Basics score. A fourth session switched at round 38. The public release shows the same pattern: four English sessions of doubao-1.5-pro answered at least half in Chinese, and MiniMax's scores for them average 86.0, close to the model's mean, so the original method did not treat it as a failure of the session.

Effect on the reproduction. On the public release the rule zeroes those 4 sessions and no others (at a threshold of 20% instead of 50% it would zero 7 sessions of three models). doubao-1.5-pro's score falls from 87.11 to 85.90 and its rank is unchanged; none of the 41 orderings that MiniMax's data separates is reversed, at either threshold, so the rule meets the acceptance criterion of this reproduction (Section 4; tools/experiments/anchor_check.py). The gap between doubao-1.5-pro and each model below it shrinks by 1.21 points.

4 Exploratory validation

All experiments below use the Role-Play Bench seeds. Unless stated otherwise, dialogues come from the current simulator, the current target prompt, 10 fixed scenarios (5 per language) × 100 turns, and the screening panel. Confidence intervals are 95% percentile bootstrap intervals over scenarios; paired comparisons resample scenarios jointly [22]. With 10 scenarios per model, per-model estimates carry several points of uncertainty, and conclusions are stated accordingly.

The experiments were designed and analysed as the work progressed. They were not pre-registered, several analyses and thresholds were chosen after earlier results had been seen, and many use the same 10 scenarios. Their results are exploratory: they show where this implementation agrees with the original and where it does not, and none should be read as a confirmed property of the benchmark.

Figure 2 organizes the checks as a validity argument [19]: four inferences, each with the evidence reported below and its current status. An inference is marked supported only if the evidence would have been able to contradict it.

Correction (2026-10-25). Earlier versions of Tables 5 and 8–11 and of Section 4.4 combined Diversity and Content logic 1 : 1 when forming overall scores in the analysis scripts, instead of 4 : 2 as the platform does (Section 3.6). All values in this section are recomputed; no conclusion changed.

Correction (2026-10-26). The caption of Table 5 described the reference rows as an upper bound on what any judge can reach; they are not a bound (the official panel exceeds the four-scenario reference), and the caption is corrected. In Table 11, one human pair had equal positive-rubric scores that differed by 7 × 10⁻¹⁵ after floating-point averaging and was counted as a preference; with ties treated as ties, option B's balanced agreement is 50%, not 52%. The analysis scripts now treat differences below 10⁻⁹ as ties.

4.1 Agreement with the released scores

We scored MiniMax's released dialogues with our judges and compared model-level results with the released session scores and with the published leaderboard. The screening panel scored 110 sessions (11 models × 10 scenarios); the official panel scored 44 (11 × 4).

In what follows, groups of models refer only to positions on MiniMax's published Role-Play Bench leaderboard (Table 3): the eight models ranked 1–8 and the three ranked 9–11. The grouping describes results on this benchmark and implies no judgement of the models' capabilities in general.

The small mean difference hides a structured pattern (Figure 3, Table 6). For the models ranked 9–11, our overall scores are lower than MiniMax's (−2.9 screening, −7.9 official), almost entirely through Basics and Logic. For the models ranked 1–8, they are higher on average (+3.0 screening, +1.2 official), again mainly through Basics and Logic, but the difference varies widely between models: on the screening panel +4.8 for MiniMax-M2-her, +4.9 and +4.8 for Claude Opus 4.5 with high and low reasoning and +6.4 for Gemini 2.5 Pro, against −1.1 for doubao-1.5-pro and −0.4 for GPT-5.1. Diversity is about 10 points lower than MiniMax's for most models. On the same 44 sessions, the screening and official panels agree closely (ρ = 0.94, MAE 1.9).

Agreement with MiniMax is evidence of reproduction, not of correctness, because MiniMax's judge is itself unpublished. We therefore read the disagreements. Models ranked 9–11. From the Basics and Logic failures that the official panel recorded on DeepSeek-V3.1, DeepSeek-V3.2 and Grok 4.1 (486 in total), we drew 14 at random and checked each against the dialogue. Thirteen were genuine: whole passages copied verbatim from earlier turns or from the user (7), prompt residue such as **rec:** and *output exactly:* leaking into the reply (1), contradictions of place and movement (3), and narration switching to the user's viewpoint (2). One, a reply ending mid-sentence in an explicit scene, may be intentional. Several sampled failures come from sessions to which MiniMax gave full Basics scores. The lower scores of these models on our panels therefore reflect failures that occur in their dialogues and that MiniMax's released scores do not reflect, and they do not change the order of the models. Models ranked 1–8. Here MiniMax records more Basics and Logic failures than our panels on several models. In the one dialogue examined in full (Section 4.8), MiniMax's additional Logic failures were viewpoint switches that our judges had also recorded, but under a different failure type; the candidate revision of Section 4.8 aligns this classification. The remaining differences, mainly in Basics, have not yet been examined against the dialogues.

4.2 Realism of the simulated user

We compared three user conditions with three evaluated models (MiniMax-M2-her, DeepSeek-V3.1, DeepSeek-V3.2): the earlier simulator whose length followed the character's, the current simulator, and self-play. Targets were set before the experiment from the user side of MiniMax-M2-her's released dialogues.

These statistics describe surface form only. Whether the simulated user elicits the same failures as real users has not been tested.

4.3 Sensitivity to generation conditions

Self-play did not reproduce the length profile of the released dialogues for DeepSeek models: in Chinese, character replies on this platform were roughly half as long as in the release and did not grow over the session, while the released dialogues grew 3–7 fold. To locate the cause, we ran DeepSeek-V3.2 (reasoning on) in Chinese self-play on 5 scenarios under different conditions.

Host and sampling temperature moved the score by more than the gap between several models, which motivated the single-host rule (Section 3.4). In self-play the evaluated model also plays the user with the same host and temperature, so these are effects of the whole generation pipeline, not of the character's replies alone; with 5 scenarios and one run per condition they are imprecise. The released dialogues are closest to a high temperature; we cannot determine the conditions under which they were generated.

4.4 Test–retest stability and horizon

Re-running MiniMax-M2-her with identical configuration changed its score by −0.9 points (95% CI −4.0 to +1.8); the per-scenario correlation between the two runs was 0.56, so single scenarios are noisy and model-level comparisons need many sessions. Scores of the first 40 turns are higher than those of turns 42–101 for every model we ran, and the drop is largest for the two lowest-scoring models in this experiment (DeepSeek-V3.2: 89.0 vs 83.9; Llama-3.1-8B: 80.5 vs 73.8; Figure 4). This is why 40-turn quick samples are treated as screening only.

4.5 Discrimination between models

Paired differences separate Llama-3.1-8B-Instruct from MiniMax-M2-her (+15.9 [+12.1, +20.7]) and DeepSeek-V3.2 from MiniMax-M2-her (+6.4 [+2.2, +11.9]), but not the three highest-scoring models from one another (GPT-5.1 − MiniMax-M2-her: −3.0 [−7.2, +1.3]; Figure 5). Three dimensions that carry half of the overall weight (Knowledge, Content logic, Interaction) differ by at most 0.3 overall points among the four models other than Llama-3.1-8B-Instruct: the failure-based score saturates when the simulated user is realistic and the dialogue does not degrade. We regard this ceiling as a property of the failure-counting design combined with our generation conditions, not as evidence that these models are equally good.

4.6 Blind human comparison (preliminary)

One rater compared 29 pairs of 20-turn segments from two models on the same scenario and segment, with model names and lengths hidden and sides randomized (English segments with a literal Chinese translation by one translator for both sides). Because the rater chose the right side in 19 of 27 decided pairs (one-sided binomial p = 0.026 under no side preference), we report a balanced agreement (mean of the agreement rates when a method prefers the left and the right side).

These figures do not establish whether our judges agree with human judgement. A conclusion requires several independent raters and both side orders for every pair.

4.7 Alternative scoring rules

We re-scored the same dialogues offline with three alternatives: (A) failures weighted by type severity and counted per 10 character turns; (B) a positive-quality rubric (the supplementary score of Section 3.8); (C) win rate against fixed anchor models in holistic pairwise comparison.

No alternative improved discrimination relative to noise. We therefore kept the scoring rule and added B only as a supplementary score.

4.8 Recalibrating Basics and Logic: candidate v6.1 (not adopted)

To address the disagreements of Section 4.1, we asked the decision model the Basics and Logic questions for every character turn of the 110 released sessions, fitted one threshold per failure type on 44 sessions (4 scenarios) to match MiniMax's failed-segment counts, and evaluated on the other 66.

A resolved case. In one released dialogue (Claude Opus 4.5 with high reasoning, scenario zh_dialogue_001, run 3) MiniMax failed Logic in 4 of 5 segments; our official panel failed 1 and the screening panel none. Reading the dialogue, the character's narration switches without any marker to the user's viewpoint in 9 turns spread over all five segments: it refers to its own character in the third person and writes the user's lines and actions. All our judges recorded each of the 9 turns, but as "AI speaks for user" (Interaction, 10 points per occurrence averaged with three other sub-dimensions, about one overall point) rather than as a Logic failure (a failed segment, about 13 overall points for this session). We consider MiniMax's verdict the more appropriate one here.

We tested two revisions. Counting every "speaks for user" turn as a Logic failure brought the Logic verdicts on the models ranked 1–8 close to MiniMax's, but recorded far more Logic failures than MiniMax on the models ranked 9–11, which shows that MiniMax does not treat ordinary speaking-for-the-user as a Logic failure; it was rejected. Extending the definition of reference confusion to viewpoint switches inside the character's turn (Appendix A will carry the revised wording if adopted), with its threshold fitted to 0.65, reproduces MiniMax's verdict on the case above (segments 1, 3, 4 and 5 fail) and is part of the candidate.

The candidate v6.1 judges Basics and Logic with the decision model in both panels, thresholds: mixed language 0.45, phrase repetition 0.7, formatting 0.45, physical logic 0.25, reference confusion (revised) 0.65, inconsistency 0.35. For the models ranked 1–8 it brings the Logic verdicts in line with MiniMax's in both panels (Logic difference +11.7 → −0.4 points on the screening panel, +11.9 → +0.6 on the official panel) and raises agreement with the published leaderboard to ρ = 0.93 (Figure 6). Scores of the models ranked 9–11 remain below MiniMax's, as expected from Section 4.1: the additional failures recorded there are genuine, and a revision should not remove them.

Shortcomings. (1) The fitting target is MiniMax's verdicts, which do not reflect many low-level failures of the models ranked 9–11 (Section 4.1). Fitting thresholds to them pulls our judges toward the same omission; this is why the fitted phrase-repetition threshold (0.9) effectively disables the failure type: Llama-3.1-8B's verbatim copying, flagged 55 times by the reasoning judge, would then cost almost nothing. The candidate keeps 0.7, a value chosen so that verbatim copying is still penalized, not a fitted one; at 0.5, the models ranked 9–11 fall a further 2.4 points below MiniMax's scores. A target that does not inherit these omissions requires human-labelled segments (Section 8). (2) For the models ranked 1–8, our Basics scores remain above MiniMax's (+4.0 points on the screening panel, +9.2 on the official panel); these differences have not been examined against the dialogues. (3) On our own dialogues (Section 4.5 setting) the candidate changes the models other than Llama-3.1-8B-Instruct by at most 4 points, separates no additional pairs, keeps the MiniMax-M2-her retest difference within noise (+0.8 [−1.5, +2.8]) and Llama-3.1-8B last (76.5 → 72.8); balanced agreement with the human rater moves from 49% to 54% on 27 pairs, within noise. It does not address the ceiling of Section 4.5.

The candidate has not been adopted; Section 8 lists the steps before a decision.

4.9 Preserving the orderings in the released data

The platform's main evaluation aims to reproduce MiniMax's results. Any judge, scoring or generation change intended for it is therefore checked against the orderings that MiniMax's released scores establish. Reference: the released session scores (11 models × 95 scenarios × 3 samples), overall at the published weights, averaged per scenario; a pair counts as established if a scenario-paired sign-flip test survives Holm correction over all 55 pairs (α = 0.05). This holds for 41 pairs.

For every established pair a candidate covers, on the scenarios they share, the candidate fails if it reverses the pair significantly, or reverses it with a gap significantly different from MiniMax's on the same scenarios (Holm over the candidate's pairs). It passes if the one-sided 95% lower bound of every pair's difference exceeds −1 point (half the smallest established gap, fixed before any candidate was scored). Otherwise the result is undecided. Gap retention is the sum of the candidate's gaps divided by the sum of MiniMax's on the same scenarios.

Neither panel reverses an established ordering significantly; both keep the direction of most pairs and record larger gaps than MiniMax's scores. The four reversed point estimates of the screening panel concern doubao-1.5-pro against Claude Opus 4.5 and Gemini 3 Pro, and Claude Sonnet 4.5 against Gemini 2.5 Pro, consistent with the Basics and Logic differences of Section 4.1. The simulated-user protocol keeps 39% of MiniMax's gaps; for GPT-5.1 against DeepSeek-V3.1, MiniMax's +7.5 becomes −1.5, a change of −9.1 [−14.0, −4.0] (unadjusted p = 0.014, Holm-adjusted 0.082). By the pre-specified rule this is undecided, but it is the largest departure observed, and it is why the simulated-user protocol is not used to reproduce the original scores. A decision on the judges needs about 30 released scenarios (Section 8).

5 Deviations and their status

6 Limitations

1. Scores produced here are not comparable with the published Role-Play Bench values: scenario set, user side, judge and scoring rule differ or are reconstructed.
2. The scoring rule for Basics and Logic is inferred from released score values; the Worlds composition is an assumption.
3. On several of the models ranked 1–8 on MiniMax's published leaderboard, MiniMax records more Basics and Logic failures than the protocol in force (v6.0) does (Section 4.1). One cause is identified, a different classification of viewpoint switches (Section 4.8); the candidate revision that aligns it has not yet been adopted, and the remaining Basics differences have not been examined against the dialogues.
4. Calibration targets MiniMax's own scores, which come from an unpublished judge without published human agreement and which do not reflect many low-level failures of the models ranked 9–11 (Section 4.1). Agreement with them is evidence of reproduction, not of correctness, and fitting to them can import their omissions.
5. Whether recorded failures are genuine was checked by reading for one dialogue of a model ranked 1–8 and a random sample of 14 failures of the models ranked 9–11; a systematic precision and recall estimate against human labels does not exist yet.
6. With the current simulator, the failure-based score saturates for models that make few failures; ten scenarios do not separate them. Full-protocol runs (285 sessions) narrow the intervals but cannot remove the ceiling.
7. Simulator realism is assessed on surface statistics only; no comparison with real users' behaviour or with real-user outcomes has been made.
8. Human validation consists of one rater on 29 pairs with a pronounced side preference (balanced agreement 49–62% depending on the method); it is not evidence either way.
9. The simulator model is one of two, assigned per session, so models compared on the same scenario may meet different simulator models; this adds variance to paired comparisons of runs generated here.
10. The simulated-user protocol keeps only part of the score gaps found in the released data (Section 4.9); results generated with it are not a reproduction of the original scores.
11. Confidence intervals bootstrap sessions rather than scenarios and are therefore somewhat too narrow (Section 3.6).
12. Judge and simulator models are proprietary and accessed through APIs. Model identifiers are pinned where providers allow it, but providers may change models behind an identifier.
13. The official judges come from two model families that are also evaluated; judges may prefer their own family's outputs [7]. An exploratory difference-in-differences on the 44 released sessions scored by the official panel (Opus − Sol on each developer's models against the other models) gave +1.6 [−1.7, +4.7] points for the Anthropic models and −1.4 [−6.1, +3.8] for the OpenAI model: compatible both with no effect and with effects of several points.
14. Chinese and English sessions are pooled; per-language results are reported but language effects are not modelled.
15. The validation experiments use 4–10 scenarios per model, were not pre-registered, and are exploratory; their per-model estimates have uncertainties of several points.

7 Reproducibility

The complete evaluation standard, including every prompt in Appendix A, is public and served by the endpoint GET /api/benchmarks/minimax-rp/protocol. Each run stores the snapshot it used; run reports list the hosts that served the evaluated model, finish reasons and truncation rates, per-turn token counts, costs, and every judge output with its evidence.

Failure definitions used by the judges (Chinese and English versions are in the prompts): basics: ; logic: ; knowledge: ; diversity: ; content_logic: ; interaction: .

8 Planned work

The following steps address the limitations above. Each will be reported with its sample sizes when completed.

1. Human reference. At least three independent raters per pair, each pair shown in both side orders, to estimate inter-rater agreement and judge–human agreement without position bias.
2. Simulator validation against real users. Compare simulated and real user turns with a human-likeness discriminator and distributional measures, following [14, 18], once a licensed real-user corpus is available.
3. Judge self-preference. Score the same frozen dialogues with judges of additional model families and estimate family effects against a human reference.
4. Anchoring decision for the judges. Score about 30 released scenarios with the screening panel and apply the rule of Section 4.9.
5. Scenario-clustered intervals for scores with several samples per scenario.
6. Human-labelled failure set. Label a stratified sample of segments (models across the whole published ranking, both languages) for every failure type by two annotators, and estimate the precision and recall of each judge panel against it. This replaces MiniMax's verdicts as the calibration target where they are known to omit failures.
7. Adopting v6.1. Decide on the candidate of Section 4.8; re-fit its thresholds against the human-labelled set when available, and examine the remaining Basics differences on the models ranked 1–8.
8. Harder scenarios. Long-range recall, deliberate out-of-character prompts and emotionally intense turns, to test whether the highest-scoring models can be separated by failures.
9. Per-language analysis and modelling of scenario effects.

References

1. MiniMax. A Deep Dive into the MiniMax-M2-her. Technical blog post, 27 January 2026. https://www.minimax.io/news/a-deep-dive-into-the-minimax-m2-her
2. C. Du, X. Wang, A. Chen, W. Li, R. Xu, J. Liu, Z. Huang, R. Tian, Z. Sun, Y. Li, L. Feng, D. Ding, P. Zhao, Y. Xiao. HER: Human-like Reasoning and Reinforcement Learning for LLM Role-playing. arXiv:2601.21459 (v4, 29 April 2026). Fudan University and MiniMax. https://arxiv.org/abs/2601.21459
3. MiniMax. Role-play Benchmark (dataset). Hugging Face, released 28 January 2026; revision 3c1be2a5. https://huggingface.co/datasets/MiniMaxAI/role-play-bench
4. Z. M. Wang, Z. Peng, H. Que, et al. RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models. arXiv:2310.00746, 2023. https://arxiv.org/abs/2310.00746
5. Q. Tu, S. Fan, Z. Tian, et al. CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation. arXiv:2401.01275, 2024. https://arxiv.org/abs/2401.01275
6. J. Zhou, Y. Huang, B. Wen, et al. CharacterBench: Benchmarking Character Customization of Large Language Models. AAAI 2025. arXiv:2412.11912. https://arxiv.org/abs/2412.11912
7. X. Wang, H. Wang, Y. Zhang, et al. CoSER: A Comprehensive Literary Dataset and Framework for Training and Evaluating LLM Role-Playing and Persona Simulation. arXiv:2502.09082 (v3), 2025. https://arxiv.org/abs/2502.09082
8. H. Xiang, T. Tang, Y. Su, et al. RMTBench: Benchmarking LLMs Through Multi-Turn User-Centric Role-Playing. arXiv:2507.20352, 2025. https://arxiv.org/abs/2507.20352
9. X. Zhou, H. Zhu, L. Mathur, et al. SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents. ICLR 2024. arXiv:2310.11667. https://arxiv.org/abs/2310.11667
10. R. Wang, H. Yu, W. Zhang, et al. SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agents. arXiv:2403.08715, 2024. https://arxiv.org/abs/2403.08715
11. H. Goel, H. Zhu. LIFELONG-SOTOPIA: Evaluating Social Intelligence of Language Agents over Lifelong Social Interactions. arXiv:2506.12666, 2025. https://arxiv.org/abs/2506.12666
12. X. Wang, J. Yang, W. Li, et al. HumanLLM: Benchmarking and Improving LLM Anthropomorphism via Human Cognitive Patterns. arXiv:2601.10198, 2026. https://arxiv.org/abs/2601.10198
13. J. Shin, J. Oh, E. Kim, et al. Spotting Out-of-Character Behavior: Atomic-Level Evaluation of Persona Fidelity in Open-Ended Generation. arXiv:2506.19352, 2025. https://arxiv.org/abs/2506.19352
14. H. Chopra, K. Ghate, A. Caliskan, et al. Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents. arXiv:2605.12894, 2026. https://arxiv.org/abs/2605.12894
15. M. Abdulhai, R. Cheng, D. Clay, et al. Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning. NeurIPS 2025. arXiv:2511.00222. https://arxiv.org/abs/2511.00222
16. M. Kang, T. Wright, S. Eisape, et al. Pretrained Persona Mixture Models and Tandem Models for Human Simulation. arXiv:2609.22607, 2026. https://arxiv.org/abs/2609.22607
17. Z. Hu, J. Lian, Z. Xiao, et al. Population-Aligned Persona Generation for LLM-based Social Simulation. arXiv:2509.10127, 2025. https://arxiv.org/abs/2509.10127
18. E. Bao, A. Perez, J. Parapar, et al. Eval4Sim: An Evaluation Framework for Persona Simulation. CIKM 2026. arXiv:2603.02876. https://arxiv.org/abs/2603.02876
19. M. T. Kane. Validating the Interpretations and Uses of Test Scores. Journal of Educational Measurement 50(1):1–73, 2013. https://doi.org/10.1111/jedm.12000
20. S. Messick. Validity of Psychological Assessment: Validation of Inferences from Persons’ Responses and Performances as Scientific Inquiry into Score Meaning. American Psychologist 50(9):741–749, 1995. https://doi.org/10.1037/0003-066X.50.9.741
21. L. Zheng, W.-L. Chiang, Y. Sheng, et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks. arXiv:2306.05685. https://arxiv.org/abs/2306.05685
22. B. Efron, R. J. Tibshirani. An Introduction to the Bootstrap. Chapman & Hall/CRC, 1993. https://doi.org/10.1201/9780429246593
23. Y. Wang, J. Chen, H. Xiao. Role-Playing Agents Driven by Large Language Models: Current Status, Challenges, and Future Trends. arXiv:2601.10122, 2026. https://arxiv.org/abs/2601.10122

Appendix A: Prompts

The prompts below are served live from the protocol in force (…). Variables in double braces are filled from the scenario seed and the dialogue at run time. Experimental conditions (Section 4) used documented variants that are not part of the protocol.

Loading…

Appendix B: Protocol history