GSOC 2009 - Embodiment Language Comprehension

From OpenCog


Embodiment Language Comprehension

Here you will find all the available information about the (in development) GSoC'09 project which aims to integrate RelEx and SegSim into Embodiment. The objective of this project is to prepare Embodiment (an OpenCog module designed to instantiate a "physical body" [real or virtual] into a given host application) to receive and interpret Natural Language sentences (english), besides to answer simple questions and eventually spontaneously talk to avatars using SegSim.

A more detailed working progress log can be found here.

Integrating RelEx into Embodiment

We're using a virtual world called Multiverse to evaluate the OpenCog embodiment module. A proxy, written in Java, was developed to provide an interface between Multiverse and Opencog. Given that RelEx is also written in Java, it will be integrated into Multiverse proxy as a library.

In Multiverse, commands are given by the avatars through a console. So, I will write a code to intercept all the text commands and send it to a RelEx parsing step, if necessary, before sending it to OPC (Cogserver).

Here are a list of tasks to be executed in order to achieve the complete Embodiment/RelEx integration:

  • Proxy Multiverse
    • Create a module to intercept avatar commands (Done)
      • Detect if the given command is a question, fact or an specific command(Done)
        • Specific commands are those that will not be parsed by RelEx.
      • Implement a service to send sentences to RelEx and pack the result an Instruction for OPC (Done)
    • Prepare a list of Frames, from framenet, corresponding to the agent perceptions (Physiological needs, emotions, feelings, spatial relations, etc.)
  • OPC
    • Change PAI to receive and process the new Instruction message format (Done).
    • Add support to managing Frames in Embodiment (Done)
    • Create a Frame representation for already implemented embodiment perceptions (Done)
    • Specify the reference resolution rules to be used with RelEx inputs.
    • Implement the reference resolution rules inside the OPC.

The new instruction message format has:

  • Attribute content-type of tag instruction: Determine the type of the instruction. (QUESTION, FACT or SPECIFIC_COMMAND).
  • Attribute target-mode of tag instruction: To what OPC operation mode (PLAYING, LEARNING, SCAVENGER_HUNT, etc.) is that instruction for.
  • Tag sentence: the original text of the avatar's instruction
  • Tag parsed-sentence: output of the parsed sentence (by RelEx)

Examples of messages:

<?xml version="1.0" encoding="UTF-8"?>
<pet:petaverse-msg xmlns:pet="" xmlns:xsi="" xsi:schemaLocation=" BrainProxyAxon.xsd">
<instruction pet-id="816951" avatar-id="779547" timestamp="2009-05-29T18:29:38.203" content-type="QUESTION" target-mode="ANY">
<sentence length="18">is that your best?</sentence>
<parsed-sentence length="17099"><![CDATA[(ReferenceLink (stv 1.0 1.0)
   (WordInstanceNode "is@3b649d80-2f53-434c-80a2-2c4b46a4e308")
   (WordNode "is")
(WordInstanceLink (stv 1.0 1.0)
   (WordInstanceNode "is@3b649d80-2f53-434c-80a2-2c4b46a4e308")
   (ParseNode "sentence@4b198d05-d992-4829-8d77-1f02454f35ca_parse_0")
(ListLink   (AnchorNode "# New Parsed Sentence")   (SentenceNode "sentence@4b198d05-d992-4829-8d77-1f02454f35ca")); END OF SENTENCE]]></parsed-sentence>
<?xml version="1.0" encoding="UTF-8"?>
<pet:petaverse-msg xmlns:pet="" xmlns:xsi="" xsi:schemaLocation=" BrainProxyAxon.xsd">
<instruction pet-id="817751" avatar-id="779547" timestamp="2009-05-29T18:40:39.343" content-type="SPECIFIC_COMMAND" target-mode="SCAVENGER_HUNT_MODE">
<sentence length="24">lets play scavenger hunt</sentence>

Perceptions/AtomsFrames Mapping

The RelEx output used in this project will be a scheme program designed to create Frames (represented by atoms) inside the AtomTable. So, in order to make the current agent perceptions useful, we will convert them into the same RelEx output Frames representation.

Frames 2 Atoms

Integrating SegSim into Embodiment

This is the second part of the Embodiment Language Comprehension project. In this step, it will be added to the agent a mechanism that lets it to express its intentions using sentences in English. The original project was planned to be built with SegSim. However, at the first moment, we will use NLGen1 to do the service.

Basically, NLGen is a pattern matcher that uses a base of processed sentences to choose which sentence is more similar to the data given at its input. NLGen receives a sentence structured in Relex format as input, and returns a complete sentence, when the pattern matching succeed in finding one. The pattern matching process compares the unary and binary relations of the given sentence with the sentences stored in its database.

This part of the project comprises the following modules:

  • Classifier: responsible for identifying which kind of question was sent to the agent (yes/no question, internal state question, environment question, etc.)
  • Answer Finder: responsible for searching the AtomTable to determine the answer of the question
  • Frames2Relex: responsible for converting the answer (in Frame format) into the Relex output and send it to the NLGen to be processed.
  • Documentation : Sketch of the Project's documentation

The Questions page presents the accepted questions regarding objects location.

Using an external Ontology to improve Reference Resolution

Actually, Reference Resolution is limited to a list of Rules that identifies a few frames into the parsed sentence data and then select the appropriate real entity. A parsed sentence is composed by several Frames. These Frames represent the semantic value of the event described by the sentence. That Frames are generate from the sentence verbs and doesn't take into account the semantic of the nouns.

A concept ontology can then be used to identify the elements described by nouns in the sentences and complete the Reference Resolution process. So, here follows what was planned to this task:

Use WordNet ( as a concept hierarchy. Convert WordNet into atoms and load it by demand via SQL. At a first moment, only hypernym/hyponym and holonym/meronym relations will be used to infer the referred entity from a parsed sentence which contains nouns.

hypernym/hyponym is a relation of type isA
holonym/meronym is a relation of type isPartOf

Each type a Frame Entity, which designates a generic object generate from a noun in the parsed sentence, is found, a specific Reference Resolution rule will be executed to query the ontology and to determine the type of the mentioned object.

Once identified, the object SemeNode will be connected to the WordInstanceNode which describes the noun that has created the Entity Frame instance, as a normal Reference Resolution process does.