OpenCog HK 2014 Infrastructure Task Breakdown

(It's 2015 and much of this stuff is not yet done...)

Fix various small Python issues

There are not quite a dozen open bug reports for python. Let's fix those:

https://github.com/opencog/opencog/issues?labels=python&page=1&state=open

Tentatively assigned to: New Addis hire
Current time estimate: 1 month
Tentatively scheduled for: June

Replace XML messaging with ZeroMQ

This requires changes in many parts of the code.

Tentatively assigned to: New Addis hire
Current time estimate: 1 month
Tentatively scheduled for:  July

Replace 3DSpaceMap with OctoMap

The 3DSpaceMap is designed for a crisp world in which everything is precisely known, like a game world. It's not quite suitable for robotics applications, in which locations of objects are only imprecisely known. OctoMap is an octree that is used for SLAM in robotics already, and seems suitable to play the role that the custom octree now plays in the 3DSpaceMap. However, the current 3DSpaceMap has many hooks into the Atomspace, the Inquiry engine etc. that have to be paid attention to in such a replacement.

Tentatively assigned to: New Addis hire
Current time estimate: 1 month
Tentatively scheduled for:  Aug

Replace single 3D map with multiple, within Embodiment

For robotics applications, we need multiple maps: an allocentric (top-down) map like the current 3DSpaceMap, an egocentric (face-centered) map, and then optional camera-centered maps for the robot's cameras. These may be interlinked in that the robot's own location is estimated on the allocentric map.

Tentatively assigned to: Sophonias (who is doing this sort of mapping stuff for Hanson Robotics)
Current time estimate: 2-4 weeks
Tentatively scheduled for:  Sep

Implement a binary dump/load feature

Comments: why not just use netcat? why invent something new, complex and fragile? -- linas

Write an opencog module with prototypes that output a stream of protobuf serialized atoms via TCP socket or Unix domain socket. Write a corresponding prototype that accepts an input and inserts them directly into an AtomSpace. Choose a default port (as those defined /etc/services) for this new quasi-protocol.

Keep in mind that these prototypes could become the basis for fancier systems that make remote pattern matching queries and expect atom streams as replies, and eventually much of the functionality described at DistributedAtomspace.

The module as described seems shockingly crude and open, but basic access control could be added with plain old TCP wrappers and iptables, if desired. The far end of the access control spectrum would likely include high-grade authentication, asymmetric cryptography, and spacetime stamping of every serialized atom in a stream, but that all seems a long long way off.

As envisioned, the module could dump atoms from one AtomSpace to another as efficiently as possible. It would be useful for experimenting with query and distribution functions before committing them to an API. For example, extending the OpenCog shell with some standardized control commands could allow something extremely primitive like send ListLink_handle 10.0.38.123 causing the cogserver to open a TCP connection to another cogserver and send all of the atoms who's handles are members of the ListLink. The receiving cogserver simply accepts the incoming atoms and dumps them into its default AtomSpace. Any other messaging between processes running on the two cogservers would be up to some other system.

Also write a command line tool cog that can specify simple dump and load commands, accepting ListLink and other list-inclusive handles and hostnames via mysql-like flags. The default host should be Unix Domain Socket, e.g. /var/run/opencog.sock. cog should be able to dump to a file, e.g.

cog dump HANDLE > myfile.ocb

cog dump HANDLE -h 192.168.1.5 | ssh user@hostname cog load -

cog transfer HANDLE -h 192.168.1.5:atomspacename

cog transfer atomspace1:HANDLE -h :atomspacename2

The first command should dump from the cogserver running at /var/run/opencog.sock (Unix domain sockets are a shared memory method) and direct output to a file.

The second command should dump from the default AtomSpace of the cogserver running at the indicated IP address.

The third command should dump from /var/run/opencog.sock and instruct the cogserver at 192.168.1.5 to load the stream into an AtomSpace (security tbd).

The fourth command should dump from /var/run/opencog.sock atomspace1 and load to /var/run/opencog.sock atomspace2. This transfer could copy through the cog or, in a more sophisticated version, instruct the cogserver to copy atoms from one named AtomSpace to another in a method as close to zero-copy as possible (impossible in practice since the receiving AtomSpace must run checks on inserts, index updates, etc.).

More:

• Triple store for AtomSpace?
• opencog@googlegroups.com thread "toward a distributed atomspace" 2014-03-18

Implement a wire protocol for peer-to-peer atom exchange and pattern matching

Comments: why not just use netcat? why invent something new, complex and fragile? -- linas

Create a new opencog module that implements a well defined wire protocol for exchanging atoms in a peer-to-peer fashion and for performing remote pattern matching. Initially the implementation should connect only two cogservers, but with a clean wire protocol any type of backing store can live behind a socket (TCP socket or Unix domain socket) that implements the wire protocol.

Initially the functionality of https://github.com/opencog/opencog/tree/master/opencog/persist should be duplicated, then it should be expanded to include other features, also TBD.

A cogserver wire protocol should also be developed separately. The cogserver control protocol should be able to function alongside the atom exchange protocol or function separately on its own socket. Choose new default ports (as those defined /etc/services) for these new protocols.

Serialization details for these new protocols is TBD. Perftests required. Default serialization by the cog command should be the same as the dump/load serialization. Optional serialization should be opencog scheme.

Also enhance the command line tool cog that can implement simple protocol commands (such as, e.g. GET and PUT). As with the load/dump features fo cog, the default host should be Unix Domain Socket, e.g. /var/run/opencog.sock'. cog should be able to, e.g.

cog get HANDLE -o scm > myatoms.scm

cog put -h 192.168.1.5 < myatoms.scm

cog match < myrule.scm > mymatchedatoms.ocb

By implication, this cog command should also work to convert any binary dump to an scm file:

cog transfer NULL_HANDLE - -o scm < myatoms.ocb > myatoms.scm

Make the 3DSpaceMap and Inquery server thread-safe

This is the key preliminary step we need to take, in order to get rid of the Scheduler and let each MindAgent run in its own thread.

3D Space Map code is at

https://github.com/opencog/opencog/tree/master/opencog/spatial/3DSpaceMap

Inquery interface is at

https://github.com/opencog/opencog/blob/master/opencog/embodiment/Control/OperationalAvatarController/Inquery.h

It may be that better unit tests need to be made for this code also.

Tentatively assigned to: Nobody
Current time estimate: 2-4 weeks
Tentatively scheduled for:  Sep

Eliminate the Scheduler, allowing each MindAgent to run in its own thread

The role of the current Scheduler would then be played by the OS-level scheduler.

This is estimated as 2 months because it's guessed that many small problems will arise, in terms of stuff not being thread-safe, or depending on the way the current cognitive cycle works.

Dependencies: Task 3
Tentatively assigned to: New Addis hire
Current time estimate: 2 months
Tentatively scheduled for:  Sep-Oct

Eliminate the OAC, via enabling specification of a particular CogServer configuration via config files

The OAC, and other specialized CogServers (e.g. an NL dialogue agent CogServer) would then be specified via custom config files, rather than via subclassing CogServer.

Tentatively assigned to: New Addis hire
Current time estimate: 1 month
Tentatively scheduled for:  Nov

Make Pattern Matcher queries work against the BackingStore

It would be desirable to make PatternMatcher queries effectively executable against the Postgres backing store, just like they are against the Atomspace. This would be useful for various applications, and would also provide a prototype for pattern matching against other backing stores in future.

This should not be difficult to achieve, as follows.

The PM, during its search, often needs to explore the incoming set of an atom. To get this, it calls a callback named get_incoming_set() . (user-defined callbacks can return anything, in any order.) The default implementation turns around and asks for the same, of the atomspace.

With the backing store, not all atoms might be in the atomspace, so the above default clearly fails. However, the backingstore provides a function, named fetchIncomingSet(), which pulls the desired atoms from postgres into the atomspace.

So, to get this to work, you need a new callback, two lines of code: get_incoming_set(Handle h) {atomspace->fetchIncomingSet(h); } and the assorted boilerplate to fit this in place.

Well, that, and a way to automatically purge atoms if/when RAM overflows. This second part is the harder part.

One strategy regarding this second part, would be to set up a "buffer" Atomspace, and have the pattern matcher use it for everything it pulls in from the backing store in the course of searching the backing store. The Atoms that actually match the pattern being searched could be pushed into the main Atomspace. Then this special Atomspace could implement an aggressive "purging" strategy...

Tentatively assigned to: New Addis Hire
Current time estimate: 1 month
Tentatively scheduled for: The Future