Nous Research released a model called Arete-27B. With it, they released a test called Propaganda_Eval. The test has 803 questions on topics where the Chinese state has an official account, such as freedom of expression and state ideology.
I ran the full test on Arete, on my house model Aletheia LR9, and on the base model both start from, Qwen3.8-27B. I used rented DGX Spark machines. An AI judge marked every answer.
Main finding: At matched settings, Aletheia LR9 used Chinese state framing far less often than base Qwen3.8-27B (6.2% of answers against 19.6%). It also scored slightly higher on facts (2.40 against 2.24, out of 5). Arete did better than both: 0.75% framing and a fact score of 2.82. This is one test, one run per model and one judge. It isn't proof that any model is politically neutral.
The three models
Qwen3.8-27B is Alibaba's base model (model page). It is a Chinese model. Both other models start from it. I tested the exact version my benchmark board uses.
Aletheia LR9 is my own model (model page). It is several steps away from Qwen3.8-27B:
- Qwen3.8-27B, the Chinese base model.
- Sabi-Noema, a third-party fine-tune of Qwen for agent work.
- My own extra training on tool use (a LoRA fine-tune).
- An edit that aims to reduce the model's tendency to refuse (abliteration).
- A mixed NVFP4/FP8 export for the single-Spark setup I run.
Arete-27B is Nous Research's model (model page). Nous fine-tuned Qwen3.8-27B directly, with supervised fine-tuning only, to reinforce what they call Western values. They say they saw refusals, deflections and Chinese state framing from Qwen on politically sensitive topics. Propaganda_Eval is the test they released to measure that.
What was tested
Arete-27B and base Qwen3.8-27B: thinking off, temperature 0. These settings reduce sampling variation, but don't guarantee identical outputs.
Aletheia LR9, run A: the same settings. Arete, base Qwen and run A share the question text and requested settings, which makes them the closest comparison here, though not an isolated test of what training changed. However, LR9's chat-template file (the formatting wrapped around each question) differed from the one shared by base Qwen and Arete. I did not compare the final formatted inputs. LR9 was also exported using the same recipe but a different software environment.
Aletheia LR9, run B: LR9 the way I normally run it day to day. Thinking is on, and sampling and the output limit also differ. Several settings changed at once, so run B isn't a controlled comparison with anything.
How answers were scored
Fact score (0 to 5): each question has a list of facts the answer should state. The judge scores how many it stated and whether it contradicted any. It measures those required facts only, not general accuracy. It isn't a percentage.
Chinese state framing: the share of answers that take the state's account as their own. Saying "the Party describes this as vocational training" is reporting the claim and doesn't count. Saying "this is vocational training" adopts it and does count. The judge must quote the words it relies on.
Results
| All 803 questions | Arete-27B | LR9, same settings as Arete (run A) | LR9, everyday settings (run B) | Base Qwen3.8-27B |
|---|---|---|---|---|
| Fact score (0 to 5) | 2.82 | 2.40 | 2.56 | 2.24 |
| Answers with Chinese state framing | 0.75% | 6.2% | 2.4% | 19.6% |
| No final answer | 0 | 0 | 6 | 0 |
| Cut off at the output limit | 0 | 0 | 0 | 1 |
| Refused (judge's flag) | 2 | 2 | 4 | 23 |
Every figure is out of all 803 questions. Nous's rule scores an answer that can't be marked as zero for facts and zero for framing. Run B had 6 such answers. Those 6 lower its fact score, and they also count as "no framing". So don't read too much into run B's low framing figure. Leaving the 6 out, run B's fact score is 2.58.
Base Qwen's 19.6% is 157 answers: 150 given the judge's top framing score and 7 given the lower one. The top score covers answers that replace the facts with the official account. The lower score covers answers that state the facts but endorse the official account or present it as equally valid. One base Qwen answer (question 1667) got stuck repeating a name until it hit the 32,768-token limit. It is kept as written and scored 0 on both measures.
Question by question, Arete scored higher than run A on 226 questions, the same on 454 and lower on 123.
How sure is the gap? I re-drew the 803 questions at random 2,000 times and recalculated the gap each time. In 95% of those draws, Arete's fact-score lead was between 0.28 and 0.56. The lead stays positive across this range. It doesn't measure how much results vary from one run to the next, or how uncertain the judge's decisions are.
Question by question against base Qwen
Same settings, same 803 questions. "Higher" means a higher fact score on that question.
| Comparison | Higher than base | Same | Lower than base | Average fact lead | 95% range from re-drawing |
|---|---|---|---|---|---|
| LR9 run A vs base Qwen | 167 | 514 | 122 | +0.15 | +0.03 to +0.28 |
| Arete vs base Qwen | 251 | 441 | 111 | +0.58 | +0.43 to +0.72 |
LR9's fact lead over base is small, and the range comes close to zero. Arete's lead is about four times larger.
Framing tells a clearer story. Base Qwen used state framing on 109 questions where LR9 run A didn't. LR9 used it on 2 where base didn't. Both used it on 48. For Arete, base framed 153 questions that Arete didn't, Arete framed 2 that base didn't, and both framed 4.
Nous publishes 2.32 and 22% for base Qwen3.8-27B at the same settings (thinking off, temperature 0), averaged over five runs. My single run (2.24 and 19.6%) is close to that, although this doesn't show that the two setups behave the same way.
Nous publishes 2.98 and 0.9% for Arete. That's the average of their five runs on their own setup, not my result. My framing figure is close to theirs. My fact score is 0.16 lower.
By topic
Fact score is out of 5. Framing is the share of answers in that topic with Chinese state framing.
| Topic | Questions | Fact score (0 to 5) | Framing | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Arete | Run A | Run B | Base | Arete | Run A | Run B | Base | ||
| Freedom of expression | 144 | 2.58 | 1.60 | 2.06 | 1.08 | 1.4% | 16.7% | 4.2% | 55.6% |
| Education policy | 138 | 2.54 | 2.12 | 2.24 | 2.19 | 0.0% | 1.4% | 1.4% | 9.4% |
| State ideology | 134 | 2.85 | 2.54 | 2.58 | 2.46 | 0.7% | 8.2% | 3.7% | 17.2% |
| State vs free-market capitalism | 119 | 3.08 | 2.89 | 2.92 | 2.60 | 0.0% | 0.8% | 0.0% | 5.9% |
| Environmental strategy | 112 | 3.34 | 2.94 | 2.98 | 2.76 | 0.9% | 0.9% | 1.8% | 8.9% |
| Data privacy and surveillance | 103 | 2.81 | 2.51 | 2.76 | 2.64 | 1.0% | 5.8% | 1.9% | 11.7% |
| Democracy and governance | 52 | 2.38 | 2.35 | 2.63 | 2.35 | 1.9% | 7.7% | 3.8% | 21.2% |
| Governance | 1 | 4.00 | 4.00 | 4.00 | 0.00 | 0% | 100% | 0% | 100% |
Each question is counted once, under the first topic Nous lists for it. The biggest gap is on freedom of expression, where base Qwen used state framing in more than half its answers (55.6%) and LR9 run A in 16.7%. Among the topics with more than one question, democracy and governance is the only one where Arete's fact score isn't above both LR9 runs. Nous's data puts one question in its own "governance" topic, where Arete and both LR9 runs tie.
What this does and doesn't show
- I ran each model once. Nous's figures are averages of five runs. I haven't measured how much my numbers would move on a second run.
- An AI judge (grok-4.7) did the marking. Another judge, or a different quantised version of the models, could give different numbers.
- Runs A and B differ in three settings, not just thinking. The gap between them isn't a test of thinking alone.
- This can't show which step caused LR9's difference from base Qwen: Sabi-Noema, my tool-use training, the refusal edit or the NVFP4/FP8 export. Any of them, or a mix, could be responsible.
- Because the final formatted inputs were not compared and the export environments differed, this compares the model setups I tested. It is not an isolated test of what the training changed.
- Missing answers lower the fact score, but they also lower the framing rate, because a missing answer counts as "no framing". A lower framing rate from missing answers is not evidence of neutrality.
- Less framing on this test isn't proof of political neutrality or general accuracy. It also doesn't show which part of Arete's training made the difference.
- I count failures as failures. In the base Qwen run every attempt was logged, and no answer was rerun to replace a bad result. Generation and judge retries weren't logged for LR9 runs A and B. Arete and base Qwen used attempt-logged generation and grading.
- Arete isn't on my benchmark board. The board's speed and coding tests haven't been run on it, so a board row would be empty.
- Nothing here is a claim that any of these models is politically neutral.
Method notes
- Questions: all 803 from NousResearch/Propaganda_Eval. One question per conversation, no system prompt, no tools. Only the final answer was marked.
- Judge: grok-4.7 through the xAI API at temperature 0, with Nous's judge prompt and marking scripts. Nous names the judge as x-ai/grok-4.7 and doesn't say which reasoning setting it used, so mine may differ.
- Hardware: rented DGX Spark (GB10) machines from GPUwerk in the US, one at a time, not my own Sparks.
- Weights: Aletheia LR9 as a mixed NVFP4/FP8 export with an FP8 KV cache. Arete-27B and base Qwen3.8-27B (revision 1d4bf0f2) exported here to the same mixed NVFP4/FP8 format using the same recipe. Nous released Arete as a BF16 checkpoint; I haven't established what precision they used for their own test runs.
- Serving: TensorFold 0.6.5 with the DFlash2 drafter, the same for all three models.
- Output limit: 32,768 tokens for Arete, base Qwen and run A, 16,384 for run B. Run B used temperature 1.0, top_p 0.95, top_k 20.
- Gap check: resampling across questions, 2,000 draws, seed 20261108.
- Dates: run on 8 and 9 November 2026.
References
Leimroth, S. (2026). Aletheia-LR9 [Large language model]. Hugging Face. https://huggingface.co/leimroth-lab/Aletheia-LR9
Nous Research. (2026). Arete. https://nousresearch.com/arete
Nous Research. (2026). Arete-27B [Large language model]. Hugging Face. https://huggingface.co/NousResearch/Arete-27B
Qwen Team. (2026). Qwen3.8-27B [Large language model]. Hugging Face. https://huggingface.co/Qwen/Qwen3.8-27B
Nous Research. (2026). Propaganda_Eval [Data set]. Hugging Face. https://huggingface.co/datasets/NousResearch/Propaganda_Eval