Command Resolution (2009 Embodiment)

From OpenCog
Jump to: navigation, search

This page is an archived historical copy of a verbal command subsystem, implemented in the OpenCog Game-World Embodiment system, as implemented in 2009, and used in various virtual-world demos from 2009 through 2013. The code described here was removed from github in 2015, as a part of a large codebase cleanup effort.

CommandResolution (Embodiment)

Introduction

The goal of the Command Resolution process is to identify when the avatar requested a operation to be performed by the agent. Basically, a command is identified by checking the presence of specific Frame instances in the latest parsed sentence, said by the agent's owner.

Command Resolution is a process that must be executed after Reference Resolution, because it will look for SemeNodes directly connected, by ReferenceLinks, to the WordInstanceNodes that represent the Frames Elements values. To make the explanation easier, lets use the example bellow:

Scenario

Suppose that there is a red ball inside the environment, identified by id id_ball99. So, there will be the following nodes inside the agent's AtomTable:

AccessoryNode "id_ball99"
SemeNode "id_ball99"
ConceptNode "red"

ReferenceLink
   AccessoryNode "id_ball99"
   SemeNode "id_ball99"

EvaluationLink
   PredicateNode "color"
      ListLink
         AccessoryNode "id_ball99" 
         ConceptNode "red"

Now, suppose the owner has said the sentence "grab the red ball". A bunch of atoms representing the parsed sentence will be loaded into the AtomTable. Here are some WordInstanceNodes, extracted from the Relex2Atoms output loaded into the agent's AtomTable:

WordInstanceNode "grab@a6460c2d-b5f8-4287-8882-028d12de42d2"
WordInstanceNode "the@5bc9cde1-36b2-4bcb-97e8-da323175e4f9"
WordInstanceNode "red@216e8536-4867-49bc-970a-fc69608e39d2"
WordInstanceNode "ball@2227209d-89f8-444f-ab90-d749bbc71eda"

After the execution of Reference Resolution, each noun/pronoun present in the sentence will be solved. In the case above there is only one noun, so the answer will be:

ReferenceLink
   SemeNode "id_ball99"
   WordInstanceNode "ball@2227209d-89f8-444f-ab90-d749bbc71eda"

Now, the requested command will be detected by the Command Resolution process that executes a list a rules, described by ImplicationLinks, that searches into the AtomTable to find specific Frames instances that belong to the latest parsed sentence. A rule consists in preconditions and effects, as shown by the ImplicationLink bellow:

BindLink
   ListLink
      TypedVariableLink
         VariableNode   "$agent"
         ListLink
            VariableTypeNode "ConceptNode"
            VariableTypeNode "WordInstanceNode"
      TypedVariableLink
         VariableNode "$framePredicateNode"
         VariableTypeNode "PredicateNode"      
      TypedVariableLink
         VariableNode "$frameAgentPredicateNode"
         VariableTypeNode "PredicateNode"         
      TypedVariableLink
         VariableNode "$frameDepictivePredicateNode"
         VariableTypeNode "PredicateNode"         
      TypedVariableLink
         VariableNode "$frameEntityPredicateNode"
         VariableTypeNode "PredicateNode"         
      TypedVariableLink
         VariableNode "$targetEntity"
         VariableTypeNode "WordInstanceNode"         
      TypedVariableLink
         VariableNode "$targetEntitySemeNode"
         VariableTypeNode "SemeNode"
               
   AndLink
      InheritanceLink
         VariableNode "$framePredicateNode"
         DefinedFrameNode "#Manipulation"            
      InheritanceLink
         VariableNode "$frameAgentPredicateNode"
         DefinedFrameElementNode "#Manipulation:Agent"            
      InheritanceLink
         VariableNode "$frameDepictivePredicateNode"
         DefinedFrameElementNode "#Manipulation:Depictive"            
      InheritanceLink
         VariableNode "$frameEntityPredicateNode"
         DefinedFrameElementNode "#Manipulation:Entity"            
         
      FrameElementLink
         VariableNode "$framePredicateNode"
         VariableNode "$frameEntityPredicateNode"            
      FrameElementLink
         VariableNode "$framePredicateNode"
         VariableNode "$frameAgentPredicateNode"            
      FrameElementLink
         VariableNode "$framePredicateNode"
         VariableNode "$frameDepictivePredicateNode"            

      EvaluationLink
         VariableNode "$frameAgentPredicateNode"
         VariableNode "$agent"            
      EvaluationLink
         VariableNode "$frameDepictivePredicateNode"
         ConceptNode "#grab"            
      EvaluationLink
         VariableNode "$frameEntityPredicateNode"
         VariableNode "$targetEntity"
            
      ReferenceLink
         VariableNode "$targetEntitySemeNode"
         VariableNode "$targetEntity"            
      ReferenceLink
         VariableNode "$targetRealNode"
         VariableNode "$targetEntitySemeNode"            

      EvaluationLink
         GroundedPredicateNode "scm:inLatestSentence"
         ListLink
            VariableNode "$framePredicateNode"            
      EvaluationLink
         GroundedPredicateNode "scm:wasAddressedToMe"
         ListLink
            VariableNode "$agent"

   ExecutionLink
      GroundedSchemaNode "goto_object_and_grabit"
      ListLink
         VariableNode "$targetRealNode"

A BindLink determines the type constraints of the VariableNodes (variables) used in the ImplicationLink. Given that the ImplicationLink will be evaluated by a PatternMatch, which look into the AtomTable to find a structure that matches with the preconditions list, This kind of VariableNodes constraints help us to filter a lot of data.

Note that the ImplicationLink connects two parts: preconditions, listed under an AndLink, and the effect of the rule, the ExecutionLink that connects a GroundedSchemaNode. During the matching process, if all the preconditions were satisfied, the ExecutionLink will be created, and so, what means that a new requested command was solved. This solved command is then sent to the RuleEngine, as a 'requested by avatar command' to be executed.

You may noticed that there are two GroundedPredicateNodes in the preconditions list. The PatternMatch will not look into the AtomTable to find this type of structure. GroundedPredicateNodes are just filters evaluated by the end of the matching process to determine if a candidate answer is really valid or not. The program, specified in its name (i.e. "scm:inLatestSentence" is a Scheme function), evaluates the list of arguments (listed in the ListLink) and returns a TruthValue that will be used by the PatternMatch to eliminate or not candidates. More than on candidate can be found in the latest parsed sentence, which means that the owner requested more than one command in the sentence.

As mentioned before, a requested command is detected by checking the presence of specific Frame instances into the latest parsed sentence. The preconditions shows that. Note that it is looking for a instance of the #Manipulation frame to determine the the grab command was requested. After parsing "grab the red ball", Relex interprets the answer and introduce some Frame instances in the resulting output to represent the semantic value of the sentence. Here is an example of a Frame instance that is sent to the agent after the avatar say "grab the red ball":

InheritanceLink
   PredicateNode "grab@a6460c2d-b5f8-4287-8882-028d12de42d2_Manipulation"
   DefinedFrameNode "#Manipulation"
InheritanceLink
   PredicateNode "grab@a6460c2d-b5f8-4287-8882-028d12de42d2_Manipulation_Agent"
   DefinedFrameElementNode "#Manipulation:Agent"
InheritanceLink
   PredicateNode "grab@a6460c2d-b5f8-4287-8882-028d12de42d2_Manipulation_Depictive"
   DefinedFrameElementNode "#Manipulation:Depictive"
InheritanceLink
   PredicateNode "grab@a6460c2d-b5f8-4287-8882-028d12de42d2_Manipulation_Entity"
   DefinedFrameElementNode "#Manipulation:Entity"

FrameElementLink
   PredicateNode "grab@a6460c2d-b5f8-4287-8882-028d12de42d2_Manipulation"
   PredicateNode "grab@a6460c2d-b5f8-4287-8882-028d12de42d2_Manipulation_Agent"
FrameElementLink
   PredicateNode "grab@a6460c2d-b5f8-4287-8882-028d12de42d2_Manipulation"
   PredicateNode "grab@a6460c2d-b5f8-4287-8882-028d12de42d2_Manipulation_Depictive"
FrameElementLink
   PredicateNode "grab@a6460c2d-b5f8-4287-8882-028d12de42d2_Manipulation"
   PredicateNode "grab@a6460c2d-b5f8-4287-8882-028d12de42d2_Manipulation_Entity"

EvaluationLink
   PredicateNode "grab@a6460c2d-b5f8-4287-8882-028d12de42d2_Manipulation_Agent"
   ConceptNode "#you"
EvaluationLink
   PredicateNode "grab@a6460c2d-b5f8-4287-8882-028d12de42d2_Manipulation_Depictive"
   ConceptNode "#grab"
EvaluationLink
   PredicateNode "grab@a6460c2d-b5f8-4287-8882-028d12de42d2_Manipulation_Entity"
   WordInstanceNode "ball@2227209d-89f8-444f-ab90-d749bbc71eda"

The representation above describes a Frame instance which identifies that who said the sentence wants the agent (#Manipulation:Agent = #you) to execute a grab (#Manipulation:Depictive = #grab) on a ball (#Manipulation:Entity = ball@2227209d-89f8-444f-ab90-d749bbc71eda). But wait, doesn't the avatar wants a Red Ball, does he? Ok, don't worry, note that the WordInstanceNode which represents the ball was already solved by Reference Resolution and a SemeNode that represents a red ball was attached to it. So the goto_object_and_grabit argument will be a red ball.

Well, after a command was detected by the Command Resolution process, a requests is then made to the Agent's interface. Is there a Rule "executeRequestedAction" available in the RuleEngine which check, at each RuleEngine cycle, if is there a command request do be solved. Once there is a request to be solved by the agent, the RuleEngine then add the requested Combo Schema (action) as a suggested action. If, in the current RuleEngine cycle there is no other suggested action with a strength greater than the requested action, then the request will be executed by the agent, finalizing the whole Command Resolution process.

Adding more commands

Each ImplicationLink, which describes how to identify a specific requested command, can be found in the Scheme file opencog/embodiment/scm/command-resolution-rules.scm . If you want to add a new one, write an ImplicationLink which contains, in the effect side, an ExecutionLink surrounding a GroundedSchemaNode and define a variable pointing to its body. So, add that variable to a list named 'command-resolution-rules' at the end of the file and voila! the new command should be solved in the Resolution process. Ok it isn't that simple. Before anything you must to calibrate the Relex rules to detect the command you're working on and check if it is sending Frames instances that represents the command. The preconditions of your ImplicationLink must match the exactly shape of the Frames instances Relex is sending to the agent. Finally, the GroundedPredicateNode must be named with a real and tested Combo Schema. That name will be used by the RuleEngine to execute a Combo Schema using the SchemaRunner/ComboInterpreter.

ExecutionLink
   GroundedSchemaNode "walkto_obj"
   ListLink
      VariableNode "$targetRealNode"

In the example above 'walkto_obj' is a real Combo procedure that receives just one element as argument and makes the agent to move to the position of the element. The VariableNode must be bound to a real (which represents some real object into the environemtn) node as an AccessoryNode, StructureNode, ObjectNode, etc.

If both Relex send the correct Frames Instances in its output and the ImplicationLink was configured correctly (both pre-conditions and effect), you can start sending commands to the agent, just using english sentences.

Implemented Commands

Here follows the list of the implemented commands:

go to: make the agent to move from one point to another.
.i.e. 
"go to the red ball" will make the agent go to the position of the nearest red ball.
grab: make the agent to grab a pickupable element.
.i.e. 
"grab the red ball" will make the agent go to the position of the nearest 
   red ball and grab it.
"grab the tree" will fail because a tree is not a pickupable object, so the
   agent will ignore the command.
drop: make the agent to drop the object which is being held by it.
.i.e. 
"drop the red ball" will make the agent to drop the red ball which it is holding.
"drop it" it will produce the same result as the example above if the previous sentence, 
   said by the avatar who is sending the commands, refers to the red ball.
   like "grab the red ball" and then "drop it". 
   (Look for Reference Resolution) for more details.