What it took to make an AI-graded mood dial make sense

On 10 November the Mood dial on my signals page said neutral. Under it, the tally said 26 good stories, 5 neutral and 12 bad. Anyone reading that expects the needle well over to the right. It sat 7 degrees past centre, inside the neutral band. The maths was right. Even so, the page misled people, because the tally counted stories and the needle doesn't work from a count.

The dial is the last step of a pipeline where a model grades every news story I post, a checker reviews the grade, and the site turns 123 graded stories from the last 30 days into one reading. Most of the decisions that made that reading mean something were about the grades and the page.

Each grade is for a named reader

DeskMA is a small C# tray app of mine that finds, fact-checks and posts news to the signals page in streams: AI models and platforms, AI security, engineering, legacy modernisation and a few more. Each stream has a one-line description of its audience. When DeskMA posts a story, the Claude Code CLI, running headless on Sonnet, gives it two grades. Direction runs from -2 to 2: two steps of bad news, neutral, and two steps of good. Magnitude runs from 1 to 10 on the stream's own measure of impact. The grader also writes one line saying why it chose the direction, for the checker to read.

The grader has to be told who the news is good for. A price cut on a model is good news for a team building on it and bad news for the vendor's competitors. So the prompt names the audience, and the last line of the scale it's given says "Judge good or bad for this reader, not for the world."

The five labels and what each one means live in one C# class, which writes the scale into every prompt that needs it through a placeholder, and the site keeps the same five labels in one TypeScript file. That way a prompt can't drift from the labels the code checks. It matters because the wording of a rubric affects what a judge decides. In a paper from August 2026, classifiers trained on rubric text alone, without ever seeing the response being judged, predicted LLM judges' outputs with better than chance accuracy. The first labels were headwind and tailwind. I replaced them on 7 November with plain good news and bad news, which need no key.

The checker can clear a grade but never change it

The run that fact-checks each story's claims answers one more question per story: does the proposed direction hold up for this reader, given what the pages say? It answers true, false or null. It isn't allowed to suggest a different grade. True and null change nothing. On false, both grades are cleared and the story posts without them. The judge in front of my landmark guide bot can hold a reply back. This checker can't hold a story back, because a wrong grade matters less than a wrong reply.

I set it up that way because the grades are the least important thing DeskMA produces. A story that passed the fact-check is worth posting whether or not anyone agrees on its direction, so nothing in the grading path can hold one back. If the checker could rewrite grades I'd have a second grader and nothing checking that one. With only a veto, I can see what it did and count how often it did it. The grades are also protected in the database. One Postgres function writes them, and it won't overwrite a grade that's already there, so if two runs grade the same story at once the first write stands.

Between 7 and 11 November I graded the backlog in 33 runs, five stories to a chunk. The checker passed 200 grades and cleared 6.

Four of the six had the same fault. The grader's own reason said the story changed little for the reader, and its label said good news or bad news anyway. A preview of a consumer agent behind a waitlist was graded good news for teams building on the models, with a reason saying it mattered little to them. The checker read the reason and said it pointed to neutral. Of the other two, one was a model released only to cyber defenders with no date for developers, graded good news. The other was a flaw in official SDKs that could send OAuth credentials to an attacker's server, graded good news because patches were out.

So the grader leans away from neutral, even when the reason it writes says neutral. The checker caught these because it had the reason to compare with the label, and it didn't need to know anything the grader didn't. A paper from January 2026, RULERS, rests on a similar idea: judges that had to follow a fixed rubric and cite their evidence agreed with human scores better than the baselines in most of the settings it tested.

The checker has a weakness I haven't fixed. It runs on Sonnet, like the grader, with its own prompt and its own session but the same model underneath. Anthropic's guidance on building evals says it's generally best practice to grade with a different model from the one that produced the output, and the December 2025 study by Mahbub and Feng found judges prefer text they recognise as their own. So a checker on the same model as the grader probably passes some grades a different model would question.

And 6 in 206 only covers what the checker caught. How many wrong grades it let through is a different number, and the log doesn't have it.

The counts were the misleading part

The formula explains the 10 November reading. A story's score is its direction times its magnitude, and the mood is the total score over twice the total magnitude:

score = direction × magnitude          // -20 to 20
mood  = Σ score / (2 × Σ magnitude)    // -1 to 1

So the needle only reaches an end of the arc when every story sits at that end of the scale, and big stories count for more than small ones. The 26 good stories were worth 126 points and the 12 bad ones 92. The 5 neutral stories added magnitude and no score. Weighted, that's a small lean to the good, which is what the needle showed. The tally was counting stories, and the needle doesn't go by how many there are.

I didn't change the formula. Weighting by magnitude is the point of it: a credential leak in an official SDK should count for more than a minor release. I changed the card. The arc now shows the five label bands, so a reader can see the needle sitting inside a band called neutral, and how far it has to travel to reach either end. Each count in the tally has its points beside it, so the reader is looking at the same figures the needle uses. One line under the dial says big stories count for more.

If I put another AI-graded total on a page, I'd show what went into it next to it, in the same units as the total. Every grade behind this dial could have been right and the tally would still have misled people.

62 small stories and a needle that stayed put

The test of that came the next day. Before DeskMA, AI stories reached the site from an older feed synced from another site of mine: 156 of them between 21 September and 29 October, none graded. On 11 November I let the same grader and checker work through them, and added a switch to the Mood card between the last 30 days and all time, because the graded stories now reached back to September.

In the 30-day window the backfill doubled the graded stories, from 61 to 123, and far more of the tally was neutral. Before it, 8 of the 61 were neutral. After it, 54 of 123 were. The synced AI stories came back neutral 46 times in 62. The AI stories DeskMA had picked itself came back neutral twice in 12, from the same grader and prompt.

My reading is that a different editor picked the synced stories for a different audience, so for this reader most of them change little. These numbers can't prove that.

The needle barely moved. With the synced stories the 30-day mood is -0.029, and without them it's -0.028. Their magnitudes averaged 2.2, against 5.6 for DeskMA's own stories, so 62 stories added about 40 per cent to the total weight and almost nothing to the score. What kept the dial steady was the weighting, the same thing that had confused readers the day before.

The AI stream's own bar did move, from 43 points of good news against 21 of bad to 67 against 54. That's the change a reader of the AI stream would see, and it came from the older stories I'd added to the sample. That week's news hadn't changed. When the ONS took UK earnings growth apart for the first quarter of 2021, 5.6 of the 7.8 points of growth came from a change in who was in the workforce, which the ONS calls a compositional effect.

The 30-day view will lean on the synced stories until about 28 November, when the last of them leave the window. I put the all-time view beside it. A batch like this one shows up clearly in the 30-day window and makes much less difference across the whole history.

What I'd want to see before a team ships one

If a team brought me a design for an AI-graded number on a page, the first thing I'd ask is what could move the number other than the news itself. Here two things could. The grader leaned away from neutral, and the checker caught that by holding each label to the grader's own reason. When the synced stories arrived the sample changed, and the only check for that was looking at what went into the window before reading the change.

DeskMA has no person checking grades. The log counts the grades the checker cleared and has no figure for the wrong ones it passed. To get that figure I'd read a weekly sample of 20 passed grades by hand against their sources.

Sources

• Judging LLM-as-a-Judge: Concerning Rubric Artifacts in LLM-based Automated Text Generation Evaluation, arXiv, https://arxiv.org/abs/2609.02942

• From Rubrics to Reliable Scores: Evidence-Grounded Text Evaluation with LLM Judges, arXiv, https://arxiv.org/abs/2601.08654

• Define success criteria and build evaluations, Anthropic, https://platform.claude.com/docs/en/test-and-evaluate/develop-tests

• Mitigating Self-Preference by Authorship Obfuscation, Mahbub and Feng, arXiv, https://arxiv.org/abs/2512.05379

• How furlough and changes in the employee workforce have affected earnings growth during the coronavirus (COVID-19) pandemic, UK: 2020 to 2021, Office for National Statistics, https://www.ons.gov.uk/employmentandlabourmarket/peopleinwork/earningsandworkinghours/articles/howfurloughandchangesintheemployeeworkforcehaveaffectedearningsgrowthduringthecoronaviruscovid19pandemicuk2020to2021/2022-04-29