Unity3D OpenCog Game World and Learning Environment

From OpenCog


For an overview of the environment, see the blog. This page is mostly an overview of the internal design, and specifically about the connection to OpenCog. For progress, see the Unity Integration Testing.

Unity3D hierarchy

The project hierarchy is expected to follow a particular pattern. The top level consists of (at least) these five transforms (transforms are what Unity3D calls any nodes in the hierarchies tree).

  • Avatars - all agents, controlled by OACs or by human players, should be placed in this transform. They should also be tagged with "OCA" except the player, which should be tagged - funnily enough - with "Player".
  • Objects - all objects within the world, that OpenCog should know about, should be placed here by default. They should also be tagged with "OCObject". Objects may be relocated in the hierarchy however (e.g. if an agent picks an object up, it's usually put under that agent's transform).
  • Console - the console is an interactive console accessed by pushing '~'. This is also good for regaining a mouse cursor. Console command scripts can be added to add new commands to the console.
  • HUD - A transform to represent the overlay the player sees with detail about the world.
  • World - Generates/Loads and stores the block world landscape.

Connection with OpenCog

The communication layer of Unity embodiment learns from Multiverse proxy, but with something different. In previous architecture, the proxy runs as the central coordinator of all pets(avatars) and is responsible for maintaining their informations. Once the proxy is not robust enough to handle the exchanging tasks, then all avatars are likely to crash. In Unity version, we adopt the idea of autonomy in which each avatar takes responsibility to maintain the connection with its own OAC.


OCConnector is the basic networking component of an avatar, which takes charge of the communication with OC embodiment system. OCConnector is the derived class of NetworkElement, which is a CSharp porting from Multiverse proxy and follows the same standard. Data communications between OCConnector and OAC still adopts messages in XML format, but eventually to be replaced by protocol buffer which serializes the data in a much more efficient and elegant way.

OCConnector maintains the following message pipelines:


  • avatar-signal: signals that report both the physiological information generated by OCPhysiologicalModel and the action execution results.
  • map-info: information that contains the collected perception data from virtual world.
  • tick message: dummy messages that keep ticking the OAC to ensure its cycle updating.


  • action-plan: action plans that are received from OAC and will be sent to OCActionScheduler.
  • emotional-feeling: emotional feelings that are received from OAC and will be dispatched to virtual world interface in order to be expressed.
  • availability/unavailability or other network elements: Warning, those messages are sent on another socket (called the control socket by the router) but listened on the same port.


OCPerceptionCollector is the interface of avatar to sense the world and collect useful information to meet the demand of PAI. Currently, two categories of information would be perceived:

  1. Physical data such as size(width, length, height), color, texture and material
  2. Geographic data such as position, rotation and velocity

As in Unity environment, we construct a minecraft style world, the terrain information(blocks) should also be collected and represented in OAC. We now treat the blocks as normal objects, but since the amount of blocks would be much larger than normal types, we implement a filter to distinguish them in OAC.

Finally, all the collected data would be wrapped in map-info instances and sent to OAC by OCConnector.


OCActionScheduler is the coordinator of action sequences by dispatching the actions to action manager and performing them. The actions are placed in a pipeline and executed. When a new action plan with higher priority arrives, previous action plan would be suspended and dropped. The scheduler records the plan id and sequence number of each action, and will report to OAC the execution result of actions once done.

There is an action map in action scheduler, which mapping the action name from OAC to specific function in Unity. For example, OAC sends an action named "walk", but in Unity end, the "walk" action might be achieved by "MoveToCoordinate" or "MoveToObject" functions. In addition, actions can be divided into two types, one is built-in actions, the other is learned actions from the environment. When executing an action, the corresponding function would be retrieved as follow.

public void executeAction(MetaAction action)
    string actionName = action.Name;
    // Check if action name has been registered as built-in or learnt one.
    if (builtinActionToMethodMap.ContainsKey(actionName)) {
        string methodName = builtinActionToMethodMap[actionName].ToString();
        ArrayList args = action.Parameters;
        this.AM.doAction(gameObject.GetInstanceID(), methodName, args, 
    } else if(learntActionToMethodMap.ContainsKey(actionName)) {
        string methodName = learntActionToMethodMap[actionName].ToString();
        // Retrieve action parameters.
        ArrayList args = action.Parameters;
        ActionTarget target = null;
        foreach (System.Object arg in args) {
            if (arg.GetType() == typeof(ActionTarget)) {
                target = (ActionTarget)arg;
        if (target == null) {
            // Action target is null, notify failure.
            actionComplete(new ActionResult(null, ActionResult.Status.FAILURE, AV, args));
        this.AM.doAction(target.id, methodName, args, actionComplete);
    } else {
        // Action is not registered, notify failure.
        actionComplete(new ActionResult(null, ActionResult.Status.FAILURE, AV, action.Parameters));


OCPhysiologicalModel emerges the internal dynamics of avatar, such as energy, fitness, thirst, hunger and so on. It follows the OpenCog feelings design and interacts with OAC through OCConnector.

Action Manager

The Action Manager is a generalized system to control what actions each agent can perform at a given moment, and to provide asynchronous feedback to the initiator of an action.

Each agent has their own ActionManager, but they all share a "Event completed" event to allow components to monitor for any action that are completed. All scripts that provide an action that is to be accessed by the ActionManager (and thus are available to agents/the player/OpenCog) must inherit from OCBehaviour.


To create a new action, you need to define and override: AddAction, RemoveAction. As an example. Here is the outline of a hypothetical Consume OCBehaviour:

public class Consume : OCBehaviour {
  private ActionSummary consumeAction;
  void Start () {
    AnimSummary animS = new AnimSummary("eat");
    consumeAction = new ActionSummary(this, "Consume", animS, null, true);

  // When an Avatar/Player gets near enough, this function is called...
  public void AddAction(Avatar avatar)
     ActionManager AM = avatar.GetComponent<ActionManager>() as ActionManager;

  // When an Avatar/Player moves too far away this function is called...	
  public void RemoveAction(Avatar avatar){
     ActionManager AM = avatar.GetComponent<ActionManager>() as ActionManager;
     AM.removeAction(gameObject.GetInstanceID(), "Consume");


  // This is called when the action is initiated by Avatar a.
  public void Consume(Avatar a, ActionCompleteHandler completionCallback) {
     if (completionCallback != null) {
        // Give the callback details about the action result
        ArrayList pp = new ArrayList();
        ActionResult ar = new ActionResult(consumeAction, ActionResult.Status.SUCCESS, a, pp, "I ate some food!");


There are some extra details, which are best checked by looking at some examples (like the real "Consumeable" OCBehaviour).