• Dave Schwartz
    364
    deTERMINATOR: Confidence Level vs. Score Problem

    I thought some of you guys might find this interesting.

    • The way The deTERMINATOR works under the hood is that it has created and optimized about 600 systems.
    • When it handicaps it uses its rule set to sort the systems into an Order of Preference (OOP) by score.
    • It also creates a Level of Confidence (LOC) for the highest-ranking systems.
    • This is done uniquely for each race that it handicaps.
    • The LOC is determined by the AI based upon a set of internally developed weights considering things such as success rate in its own mind (so to speak) and sample size.

    Here's where a problem developed.
    The system uses a reward & penalty system - an internal economy, if you will.

    This could best be described as SUCCEED & BREED vs. FAIL & DIE.

    (Yes, men; it's all about sex.)

    The problem was that the AI learned that if it gave each race a low confidence level it would not be held as accountable for bad picks!

    I actually saw this problem about 30 years ago when I was writing what I called Neural Net #4.
    That Neural Network had an output that represented it's confidence level on a particular bet.

    It quickly learned that IF IT DOESN'T MAKE ANY BETS, IT WON'T DIE!


    This was actually one of the main reasons that I stopped writing Neural Networks.

    In DET I demanded that no more than a small percentage of races could be tossed for lack of confidence. I believe it was set at 6.5% during the training process.

    Any Ant that skipped more was summarily executed on the spot. (Yes, I am a heartless bastard with digital ants.)
    _____________________

    Now to the actual problem in deTERMINATOR.

    The AI compares the confidence level for its top systems and rejects the ones with a LEVEL OF CONFIDENCE that is below its minimum.

    It then begins going down from its top pick until it finds one that qualifies.

    If it had to go down too far from the top of its 600 systems to find a qualified system, it would REJECT the race and simply REFUSE TO HANDICAP IT.

    Now, the user can still handicap it manually, but I didn't foresee the ramifications.

    Imagine that you're a pick 4 player and, right in the middle of the card is a race that is pivotal in writing your tickets and you've lost your primary approach!
    _____________________

    We quickly learned that the races it targets are maiden races on the turf - especially for lightly raced but somewhat older horses. (2-year olds and young 3-year olds are generally not an issue - don't know why. Perhaps they are easier to handicap because breeding means more?)

    The driving force behind the low LEVEL OF CONFIDENCE is probably sample size. There simply aren't that many races for it to look at in this category spread throughout the year. (Time of year can be a factor as well.)
    _____________________

    The Solution
    There is a limit to how far down from best that it can go to select its system. This number is chosen by the AI - remember Succeed & Breed vs. the unpleasant alternative.

    So, I forced it to go south looking for a play.

    Never considered the fact that there might be some races where the entire system list was low confidence! (It actually was limited to going down a maximum of 21% of the systems.)

    I mean, really? Not a single pitifully bad system that qualified?

    In the end the solution was to take the top 21% of the systems and build a new sorting metric that included both confidence and score together and have the AI pick the BEST.
  • Jim Pommier
    81
    Here's what I'm seeing. I want to make sure that I'm doing this correctly.
    1. The first pop-up message: Loc: Glo_Filter_Name=NM-sp-s-turf. (17) Abort or Ignore. I select Ignore.
    2. The next pop-up message: The AI Engine WILL handicap this race. Warning: Low Confidence Threshold. I select OK.
    3. I then see at the bottom of the screen "LOW CONF" and "1". If you click the "1" you get "When you change the default Low Confidence System the race must be closed & reopened so the AI can re-handicap. The new pop-up is now: Loc: Glo_Filter_Name=Y-sp-s-turf. (21). When I reopened the "1" is now a "2".
  • Dave Schwartz
    364
    Dang.
    Upgrade in 15 minutes
  • Dave Schwartz
    364
    It's working but I left in a stop.
    I'll explain all of this tonight.

    But Upgrade coming to get rid of the UNNECCESSARY stop.
  • Jim Pommier
    81
    Thanks. I'll be on the call tonight.
  • Dave Schwartz
    364
    35 fixes it and is in your DropBox.
bold
italic
underline
strike
code
quote
ulist
image
url
mention
reveal
youtube
tweet
Add a Comment