From OpenCog

Ubergoal Dynamics

In the early phases of a OCP system's cognitive development, the goal system dynamics will be quite simple. The ubergoals are supplied by human programmers, and the system's adaptive cognition is used to derive subgoals. Attentional currency allocated to the ubergoals is then passed along to the subgoals, as judged appropriate.

As the system becomes more advanced, however, more interesting phenomena may arise regarding ubergoals: implicit and explicit ubergoal creation.

Implicit Ubergoal Pool Modification

First of all, implicit ubergoal creation or destruction may occur. Implicit ubergoal destruction may occur when there are multiple ubergoals in the system, and some prove easier to achieve than others. The system may then decide not to bother achieving the more difficult ubergoals. Appropriate parameter settings may mitigate against this phenomenon, of course.

Implicit ubergoal creation may occur if some Goal Node G arises that inherits as a subgoal from multiple ubergoals. This Goal G may then come to act implicitly as an ubergoal, in that it may get more attentional currency than any of the ubergoals.

Also, implicit ubergoal creation may occur via forgetting. Suppose that G becomes a goal via inferred inheritance from one or more ubergoals. Then, suppose G forgets why this inheritance exists, and that in fact the reason becomes obsolete, but the system doesn't realize that and keeps the inheritance there. Then, G is an implicit ubergoal in a strong sense: it gobbles up a lot of attentional currency, potentially more than any of the actual ubergoals, but actually doesn't help achieve the ubergoals, even though the system thinks it does. This kind of dynamic is obviously very bad and should be avoided — and can be avoided with appropriate tuning of system parameters (so that the system pays a lot of attention to making sure that its subgoaling-related inferences are correct and are updated in a timely way).

Explicit Ubergoal Pool Modification

An advanced OCP system may be given the ability to explicitly modify its ubergoal pool. This is a very interesting but very subtle type of dynamic, which is not currently well understood — and which potentially could be very dangerous, in the case of an intelligent and capable OCP instance.

However, modification, creation and deletion is a key aspect of human psychology, and the granting of this capability to mature OCP systems must be seriously considered.

In the case that ubergoal pool modification is allowed, one useful heuristic may be to make implicit ubergoals into explicit ubergoals. For instance: if an Atom is found to consistently receive a lot of RFSs, and has a long time-scale associated with it, then the system should consider making it an ubergoal. But this heuristic is certainly not the whole story, and any advanced OCP system that is going to modify its own ubergoals should definitely be tuned to put a lot of thought into the process!

The science of ubergoal pool dynamics basically does not exist at the moment, and one would like to have some nice mathematical models of the process prior to experimenting with it in any intelligent, capable OCP system. (Students of advanced AGI Ethics may fairly consider the prior sentence an exercise in exorbitant understatement.)