Accuracy
How we find out whether Notum is right.
A similarity score is only worth having if you know how far to trust it. So Notum’s scoring is checked against decisions that have already been made, and a figure appears on this site only once it has passed a gate.
The gate has not yet been passed, so this site quotes no figure.
Illustrative, not our result
Solid inside the ring: a correct flag. Hollow inside: a false alarm. Solid outside: a miss. Hollow outside: a correct pass.
- A tribunal found the pair similar
- A tribunal found the pair not similar
The method: pairs with a label
The benchmark is a set of pairs of marks. For every pair, a trade mark tribunal has said in a published decision how alike the two marks are, or whether confusion was likely. That finding is the label: similar, or not similar.
Notum then scores each pair as a search for one mark against a register that holds only the other, without seeing the label. A pair counts as flagged when its score reaches a fixed threshold. The threshold is set once and is never retuned to suit the pairs.
- Where the pairs come from
- Published decisions of the UK Intellectual Property Office and of the European Union’s trade mark office. The decisions supply the labels only. Notum searches the UK register and nothing else.
- How a label is set
- From the tribunal’s own words. Most labels record how alike the tribunal found the two marks to be. Some record whether it found confusion likely. A person checks each label against the tribunal’s own record of its finding, and a borderline finding is left out.
- How the set is kept honest
- It holds close to as many similar pairs as not similar ones, so a tool cannot score well by guessing one way. A pair whose two marks are the same word is dropped, because it tests nothing.
Precision and recall
Every pair comes out one of four ways. Two measures follow from counting them.
Scroll sideways to see all four outcomes.
| Notum flagged the pair | Notum did not | |
|---|---|---|
| The tribunal found the marks similar | Correct flag Notum flagged a pair that really was close. | Miss Notum let a close pair through. |
| The tribunal found the marks not similar | False alarm Notum flagged a pair that was not close. | Correct pass Notum left alone a pair that was not close. |
- Precision
- Of the pairs Notum flagged as close, the share a tribunal also found similar. Low precision means false alarms: names marked as risky that were not.
- Recall
- Of the pairs a tribunal found similar, the share Notum flagged. Low recall means misses: close names that went through.
Both matter. A tool that flags everything never misses and is no use. A tool that flags almost nothing never raises a false alarm and misses the marks that count.
The gate a figure must pass
Before this site quotes an accuracy figure, three things have to be true.
- 1
A bar set in advance
A target for precision and a target for recall, fixed before the measuring and not moved afterwards.
- 2
A result that repeats
The same pairs and the same scoring give the same answer every time, so the figure can be checked.
- 3
The whole result
The method, a version history and any shortfall against the bar, shown together. Never a headline percentage on its own.
The gate, drawn
Where it stands
The gate has not yet been passed. This site quotes no accuracy figure.
The measured result, with its shortfall against the bar, is published on the accuracy page of the product itself.
Until a figure has passed, read every result as an indication of what is on the register and how close it is, with the reasons shown so that you can judge it yourself.
What a benchmark cannot tell you
A benchmark shows how Notum’s scores line up with findings on pairs of marks. It does not say that a name is safe. A tribunal also weighs the goods and services, the people who buy them and what the marks are known for, and Notum reads only the records in its coverage.
A clear result is never permission to use a name. For the name you decide to file, take it to a trade mark attorney.
See the working on your own name.
Early access is by invitation. Tell us the name you are weighing up and we will be in touch.