DeSTIN Redesign

From OpenCog

This page describes a hypothetical, speculative redesign/reimplementation of DeSTIN at a high level. When this redesign/reimplementation might be undertaken in practice is not clear at the time of the page's creation (initially, May 24, 2014, by Ben Goertzel).

See also New DeSTIN Redesign Proposal, written more recently than the present page. And see Deep Learning Perception in OpenCog, which is even more recent.


Broad Motivation

Deep learning architectures for visual perception are becoming a dime a dozen these days, and big companies are putting big dollars into fine-tuning them to work on big hardware. DeSTIN is not yet state-of-the-art as a deep learning based vision system. I think we could make it state-of-the-art or slightly better, with sufficient attention and fine-tuning. But to me that's not the most interesting direction.

I'm more interested in developing DeSTIN into a system that is very easy to tightly integrate with other systems, including motoric or reinforcement hierarchies, and also including OpenCog (the AtomSpace, PLN, etc. etc.). This is more of a unique pursuit than building a pure deep learning vision system.

There are also other modalities that haven't been pursued adequately in the deep learning community: deep learning for audition is much more primitive than it could be (though there are papers on this), and deep learning for haptics is almost unexplored. And the same sort of architecture can of course be used for many other sorts of data, e.g. various kinds of biological data.

Narrow Motivation (in the scope of computer vision)

Even within the narrower scope of DeSTIN as a machine vision system, the current implementation is not that easy to experiment with. Changing the network topology or the algorithm used inside a node requires some fairly confusing and bug-prone hacking. The code now is OK and works, but clearly is not made to support flexible experimentation. Partly as a result of this, experimentation with variations on DeSTIN now takes place largely with various scattered partial MATLAB implementations.

Portability to GPU / APU

Part of the reason the current DeSTIN implementation is the way it is (lots of nasty C arrays), is the goal of making it easier to port the system to CUDA (or some other GPU language, e.g. OpenCL) when the algorithm is more settled time comes to optimize. (Indeed there are a couple CUDA implementations already.) Modifying the code into a more flexible OO framework, as I'm going to suggest here, would work against the goal of making CUDA parallelization relatively straightforward.

However, it's worth noting that if DeSTIN is going to be run in tight integration with OpenCog, then a CUDA implementation may not make so much sense anyway (in the DeSTIN/OpenCog hybridization context). If OpenCog is going to contribute in real-time to DeSTIN's learning, and OpenCog runs on CPU, then having DeSTIN on GPU might not save any time, since the GPU/CPU communications might well end up being a bottleneck.

It might also be possible to use OpenCL to run a refactored DeSTIN on a heterogeneous architecture such as the AMD Fusion APU (or analogous Intel or other hardware). This enables a mix of GPU type processing and CPU type processing without a big communication bottleneck between the two (although, the GPU type processing is not as fast as on a good GPU). This would allow, e.g. the processing within a DeSTIN node to occur GPU style, but the overall DeSTIN system to operate CPU style.

Some information on APU performance is here:

The Actual Design Suggestions

Now for the meat of the page; the ideas underlying the suggested redesign, which are still at a fairly high level.

The main changes I'd like to see in the DeSTIN implementation are:

  • 1) Ability to specify the network architecture via a configuration file. Basically, the network structure should be able to be any digraph. The logic of DeSTIN doesn't depend on what the network looks like.
  • 2) Ability for the processing occurring within a DeSTIN node to be carried out via a pluggable DestinNodeAlgorithm object. So if we want to try k-means, or EM clustering, or an autoencoder or some other kind of neural net or whatever inside a node, it's a matter of writing a class subclassing DeSTINNodeAlgorithm and telling the system to use this class. In an efficient implementation, the NodeAlgorithms are what might get implemented in OpenCL to utilize the parallel processing of APU, for example.
  • 3) Ability for the belief data structure to be customized. Anything extending a DeSTINBelief class should be allowable as a belief. A Belief should fulfill certain requirements, e.g. Beliefs must lie in a linear space (so they can be scalar multiplied and added); and it should be possible to query a Belief associated with node N, to find out the weighting it assigns to any given combination of Beliefs associated with the children of N.
  • 4) There should be nice tools for exporting the DeSTIN state to an Atomspace (or a MySQL database, as another option) for persistent saving.
  • 5) If running DeSTIN together with OpenCog, it should be possible to create Atoms corresponding to DeSTIN Nodes and Beliefs.

Readers of this page who are familiar with DeSTIN are encourage to add other suggestions below:

  • Add your suggestion here
  • Add your other suggestion here!

Implementation Issues

Quite possibly, to create a DeSTIN system of this nature, it will make sense to begin a new implementation rather than re-using and modifying the current code. However, it's also possible (?) this could be done organically as a modification of the current code.

For easy compatibility with OpenCog and OpenCL, and for scalable performance with large networks, it would be best for a new DeSTIN along these lines to be coded in C++.


A gutsy and interesting implementation choice would be to actually implement DeSTIN inside OpenCog -- i.e.

  • make DeSTIN nodes be OpenCog Nodes (PerceptGroupNode?) , and the parent-child links between DeSTIN nodes be OpenCog Links (PerceptGroupContainsLink?)
  • make DeSTIN Beliefs be PredicateNodes, with inputs that are ListLinks whose entries are Beliefs
  • make a DeSTIN node algorithm a GroundedSchemaNode, whose grounding is a C++ function that acts on a PerceptGroupNode as input

This would simplify implementation in many ways,

In this DeSTINCog approach,

  • The network topology would automatically be configurable, as it could be loaded in from a Scheme file like any other set of OpeCog Atoms
  • Wrapping DeSTIN node algorithms in GroundedSchemaNodes would make them automatically "pluggable"
  • The Atomspace already has many support services for querying, saving/loading, etc.
  • Obviously, the interaction between DeSTIN and OpenCog processes would be simplified

Hmmm, I've almost convinced myself... ;-)