Deep Learning in OpenCog: Proposal

Deep Learning for Vision in OpenCog (Design Notes/Suggestions)

This page is somewhat obsolete. See Interfacing Between OpenCog and Deep Neural Nets for Vision Processing for newer ideas.

This rough draft document gives some slightly more detailed ideas about how one might do deep learning for vision in OpenCog. (first version Ben Goertzel: Dec. 3, 2015 ...)

The core approach is to use OpenCog as a toolkit for connecting together nodes corresponding to different spatial or spatiotemporal regions, and for invoking lower-level (e.g. neural network) processes associated with these nodes (where the lower-level processes are then executed outside of OpenCog). So in a sense one is using OpenCog similarly to how one uses Theano or Tensorflow (though in fact one is using it as an additional wrapper around one of these toolkits, via making the GSNs that OpenCog invokes use one of these toolkits, most likely Tensorflow initially). But one is also gaining more than one gains from a toolkit like Theano or Tensorflow, via representing one’s initial, final and intermediate knowledge structures in a general knowledge representation such as the Atomspace. This opens up many possibilities for future perception-cognition integration, and generally gives one great flexibility regarding how to do things.

NOTE: these are just ideas, there may be better ways to do these things, and improvisation and creative ideas are encouraged on the part of anyone working on this…

Some Preliminary Thoughts on Design/Architecture

See this page of rough notes on Outsourcing Array Operations From OpenCog

Some Design Ideas …

ArrayNode

See Outsourcing Array Operations From OpenCog for ideas on defining and implementing ArrayNodes for representing multidimensional arrays in the Atomspace.

RegionNode

A RegionNode should correspond to a certain region within a 3DspaceMap (though it could be a 2D or 3D region). Initially we only need to support rectangular regions, but we don’t want to wire this assumption too deeply into the infrastructure; we want people to be able to play with regions of other shapes as well, later on.

Question: What is the best way to implement RegionNode and link it to the 3DspaceMap? .

Array processor schema

This is my informal term for a GroundedSchemaNode whose input is a ArrayNode , and whose output is a ArrayNode .

For instance,

GroundedSchemaNode “DenoisingAutoencoder_345.py”

could refer to a denoising autoencoder that takes in one array and outputs another, so we might have

ExecutionOutputLink
	GroundedSchemaNode “DenoisingAutoencoder_345.py”
	ArrayNode  “3456”

(which would produce an ArrayNode ).

Array processor training schema

This is my informal term for a GroundedSchemaNode whose input is: A GroundedSchemaNode, plus a ArrayNode (or a list thereof) … and whose action is to create a new GSN that is "trained" based ont he ArrayNodes. See the example at the end of Outsourcing Array Operations From OpenCog .

One can also have schema for updating trained models. For instance, we might have

ExecutionOutputLink
	GroundedSchemaNode “UpdateDenoisingAutoencoder.py”
	ListLink
		GroundedSchemaNode “DenoisingAutoencoder_345.py”
		ArrayNode “3456”

Array Normalization

What I mean by tensor normalization is the process that takes in an array with nonnegative real entries, and outputs an array whose entries sum to 1. E.g.

ExecutionOutputLink
	GroundedSchemaNode “NormalizeArray.py”
	ArrayNode  “3456”

In a normalized array, each entry may be interpreted as a probability (though the particular intepretation depends on the context in which the array is constructed).

This is useful for getting PredicateNodes out of arrays.

Spatiotemporal Network

A spatiotemporal network is a set of RegionNodes connected by RegionChildLinks and RegionNeighborLinks. So e.g. we might say

RegionChildLink
	RegionLink “666”
	RegionLink “777”

meaning that region “777” is a child of the region “666”. The child and neighbor links specify the topology of the network.

For instance, in a 2D image processing application, where the image is divided into square sub-regions on multiple layers (e.g. the typical DeSTIN setup), the RegionNodes on all but the bottom layer would have 4 children (connected via ChildLinks), and the RegionNodes except for those on the edges would have 4 neighbors (connected via NeighborLinks).

A spatiotemporal network may be considered divided into layers, where

• Layer 1 = RegionNodes with no children
• Layer 2 = RegionNodes with only Layer 0 nodes as children
• Layer 3 = RegionNodes with only Layer 0 and Layer 1 nodes as children

Etc.

(Note that if we wanted to have multiple different spatiotemporal networks involving the same spatiotemporal regions, in the above approach we would need to construct multiple RegionNodes corresponding to the same spatiotemporal regions. This doesn’t seem a big problem at the moment, so I think the above representation is OK for now.)

There are many different ways to formally specify a spatiotemporal network. One approach would be to create SetLinks corresponding to the layers of a network, so that e.g. one could say

EvaluationLink
	PredicateNode “Layer”
	ListLink
		SpatiotemporalNetworkNode “1234”
		NumberNode “3”
		SetLink 
			RegionNode “1234”
			RegionNode “555”
			…

to specify the 3rd layer of a certain network (where the SetLink lists all the regions in the 3rd layer), and

EvaluationLink
	PredicateNode “depth”
	SpatiotemporalNetworkNode “1234”
	NumberNode “5”

to specify that the network in question has 5 layers.

Simple Deep Spatiotemporal Learning Schemes

A “simple spatiotemporal deep learning scheme” (or “deep learning scheme” for short) consists of a Spatiotemporal Network,

• all of whose included RegionNodes use the same kind of array processor schema and array processor training schema (e.g. they all use denoising autoencoders, or they all use k-means clustering, or whatever)
• all of whose included RegionNodes use the same input-gathering schema, with obvious changes for the bottom and top layers as needed
• each RegionNode is associated with a single GSN denoting the array processor schema associated with that node (e.g. RegionNode “456” might be associated with (GroundedSchemaNode “DenoisingAutoencoder_456.py” ))

An input-gathering schema is a GroundedSchemaNode whose input is a RegionNode contained in a SpatiotemporalNetwork, whose job is to build an ArrayNode for input to the array processor schema and array processor training schema corresponding to that RegionNode. E.g.

ExecutionOutputLink
	GroundedSchemaNode “BottomUpInformationGatherer.py”
	RegionNode “3456”

would produce an ArrayNode to be used as input for tensor scheme and tensor processor training schema associated with RegionNode “3456”.

So, formally, we could define e.g. the “simple 2D DeSTIN” deep learning scheme via

ConceptNode “Simple 2D DeSTIN”

with

EvaluationLink
	PredicateNode “input gatherer”
	ListLink
		ConceptNode “Simple 2D DeSTIN”
		GroundedSchemaNode “BottomUpInformationGatherer.py”

EvaluationLink
	PredicateNode “ array  processor schema”
	ListLink
		ConceptNode “Simple 2D DeSTIN”
		GroundedSchemaNode “makeDenoisingAutoencoder.py”

EvaluationLink
	PredicateNode “ array update schema”
	ListLink
		ConceptNode “Simple 2D DeSTIN”
		GroundedSchemaNode “updateDenoisingAutoencoder.py”

The ConceptNode

ConceptNode “Simple 2D DeSTIN”

simply serves as a hook to find the various dynamical functions associated with the “simple 2D DeSTIN” learning scheme.

The GSN “GroundedSchemaNode “makeDenoisingAutoencoder.py” , when run on a RegionNode “345” would create the GSN “DenoisingAutoencoder_345.py” as output and link it to the RegionNode, i.e.

ExecutionLink
	GroundedSchemaNode “makeDenoisingAutoencoder.py”
	RegionNode “345”
	EvaluationLink
		PredicateNode “ array  processor schema”
		ListLink
			RegionNode “345”
			GroundedSchemaNode “DenoisingAutoencoder_345.py”

Note that the use of “_345” (i.e. of the RegionNode’s name) in the name of the GSN is for human convenience only, for internal processing the important thing is the EvaluationLink joining this GSN with the RegionNode.

One can also define the “output array ” of a RegionNode in a Spatiotemporal Network as the array resulting from: applying an input gathering schema, then applying a array processor schema, e.g.

DefinedSchemaNode “getOutputArray”
	RegionNode $R
	ExecutionOutputLink
		DefinedSchemaNode “get  array  processor schema”
			RegionNode $R
		ExecutionOutputLink
			GroundedSchemaNode “getInputArray”
			RegionNode $R	

Conceptually, the learning associated with a deep learning scheme may be executed as follows, for a spatiotemporal network with N layers:

Repeat until bored…
	For L = 1 to N (iterate through layers)
		Iterate through RegionNodes R on layer L 
					(or process them in parallel if possible)
			Apply the input gatherer, to gather the input tensor for R
			If no  array  processor exists for R, create one 
					(using the schema associated with the network)
			If in training mode, run the  array  processor training schema on R
			Run the  array  processor schema for R

This can be written as a DefinedSchemaNode, of the general nature

DefinedSchemaNode “iterate deep learning scheme”
	ListLink
		ConceptNode $C  \\ e.g. $C = “Simple 2D DeSTIN”
		SpatiotemporalNetwork $S
	SequentialAndLink
		\\ apply the above Repeat Until Bored loop
		\\ using the spatiotemporal network $S and
		\\ the schema associated with $C
		\\ (note we need to use tail recursion here)

in such away that the same DefinedSchemaNode can be used even if one substitutes various different GSNs for the input gatherer, and the array and array processor training schema, and even if one switches from 2D to 3D, etc.

If the gathering of the input array includes parent as well as child information, then we have top-down feedback going on, meaning that running the top-level iteration over and over again may keep yielding different results.

Note that the learning here is totally localized, i.e. the learning occurs only locally within each RegionNode. This can only yield good results if the top-down feedback works well, i.e. if the input-gathering schema includes parent and neighbor information and this information is usefully incorporated in learning.

Region Properties

How can information get read out of one of these networks, for use e.g. by reasoning algorithms? One may construct predicates from normalized tensors, e.g. DefinedPredicateLink “get output property value”

	ListLink
		RegionNode $R
		NumberNode $N
		ExecutionOutputLink 
			GroundedSchemaNode “getArrayElement”
			ListLink
				ExecutionOutputLink
					GroundedSchemaNode “getOutputTensor.py”
					RegionNode $R
				NumberNode $N

would produce a probability value corresponding to the $N’th entry of the output tensor from the RegionNode $R.

So the logical output of a deep network, in this framework, is obtained from the predicatized, normalized versions of the output ArrayNodes of the various RegionNodes. Properties are of the form “the k’th property of the output tensor of the RegionNode R” --- meaning, “the k’th property that has been perceived/cognized to be currently true, as regards the region corresponding to R. “

A Simple Initial Example

As an illustrative example and first case of the above approach, it might be a good idea to make a 2D unsupervised DeSTIN-style stacked autoencoder network using the above ideas. Suppose one began with a 32x32, 2D gray-scale image. Then one would have a simple SpaceMap, effectively 2D, divided into 32x32 little squares. Input images would be streamed into this SpaceMap.

One could have

• Layer 1 = 4x4 squares, i.e. 64 RegionNodes here
• Layer 2 = 8x8 squares, i.e. 16 RegionNodes here
• Layer 3 = 16x16 squares, i.e. 4 RegionNodes here
• Layer 4 = 1 RegionNode corresponding to the whole SpaceMap

ChildLinks would be built from a RegionNode R on layer k+1 to the RegionNodes on layer k representing R’s sub-regions

NeighborLinks can be ignored for this simple example

The input of a node on Layer 1 consists of a 16-dim real vector built from its pixels’ gray-scale values (a vector being a 1D tensor…)

The input of a node on Layer k, for k>1, consists of a vector formed by concatenating the output vectors of its children.

The tensor processor schema associated with a RegionNode applies a denoising autoencoder that has been trained on the input to that RegionNode.

The tensor processor training schema associated with a RegionNode does incremental (online) updating of the denoising autoencoder associated with the RegionNode

NOTE: If there is not convenient available code for incremental training of denoising autoencoders, then we can use any other learning scheme here. It’s important that the learning be incremental rather than batch-mode though. Batch learning is not suitable for an AGI system…. The above-suggested system includes no feedback, only feedforward processing. To include feedback in the simplest way, the parent RegionNode’s output could be included as part of the child node’s input….