deTERMINATOR: Confidence Level vs. Score Problem

I thought some of you guys might find this interesting.

• The way The deTERMINATOR works under the hood is that it has created and optimized about 600 systems.
• When it handicaps it uses its rule set to sort the systems into an Order of Preference (OOP) by score.
• It also creates a Level of Confidence (LOC) for the highest-ranking systems.
• This is done uniquely for each race that it handicaps.
• The LOC is determined by the AI based upon a set of internally developed weights considering things such as success rate in its own mind (so to speak) and sample size.

Here's where a problem developed.
The system uses a reward & penalty system - an internal economy, if you will.

This could best be described as SUCCEED & BREED vs. FAIL & DIE.

(Yes, men; it's all about sex.)

The problem was that the AI learned that if it gave each race a low confidence level it would not be held as accountable for bad picks!

I actually saw this problem about 30 years ago when I was writing what I called Neural Net #4.
That Neural Network had an output that represented it's confidence level on a particular bet.

It quickly learned that IF IT DOESN'T MAKE ANY BETS, IT WON'T DIE!


This was actually one of the main reasons that I stopped writing Neural Networks.

In DET I demanded that no more than a small percentage of races could be tossed for lack of confidence. I believe it was set at 6.5% during the training process.

Any Ant that skipped more was summarily executed on the spot. (Yes, I am a heartless bastard with digital ants.)
_____________________

Now to the actual problem in deTERMINATOR.

The AI compares the confidence level for its top systems and rejects the ones with a LEVEL OF CONFIDENCE that is below its minimum.

It then begins going down from its top pick until it finds one that qualifies.

If it had to go down too far from the top of its 600 systems to find a qualified system, it would REJECT the race and simply REFUSE TO HANDICAP IT.

Now, the user can still handicap it manually, but I didn't foresee the ramifications.

Imagine that you're a pick 4 player and, right in the middle of the card is a race that is pivotal in writing your tickets and you've lost your primary approach!
_____________________

We quickly learned that the races it targets are maiden races on the turf - especially for lightly raced but somewhat older horses. (2-year olds and young 3-year olds are generally not an issue - don't know why. Perhaps they are easier to handicap because breeding means more?)

The driving force behind the low LEVEL OF CONFIDENCE is probably sample size. There simply aren't that many races for it to look at in this category spread throughout the year. (Time of year can be a factor as well.)
_____________________

The Solution
There is a limit to how far down from best that it can go to select its system. This number is chosen by the AI - remember Succeed & Breed vs. the unpleasant alternative.

So, I forced it to go south looking for a play.

Never considered the fact that there might be some races where the entire system list was low confidence! (It actually was limited to going down a maximum of 21% of the systems.)

I mean, really? Not a single pitifully bad system that qualified?

In the end the solution was to take the top 21% of the systems and build a new sorting metric that included both confidence and score together and have the AI pick the BEST.