Distributed Architecture

From OpenCog

Jump to: navigation, search

OpenCog must be able to function in a distributed environment, with many OpenCog server instances working collaboratively, and accessing a shared data repository. This page attempts to provide a basis for developing that architecture. A more concrete proposal by Joel with a request for comment is being written.

Contents

Core Assumptions

I) The core opencog systems (PLN, etc) act on atoms kept in RAM, which are managed by AtomTable. The core systems cannot/do not directly access Atoms from some remote location, but can work with AtomTable to get a local copy instantiated.

II) There should be some persistent repository for atoms. This repository can be shared among dozens or thousands or millions of opencog instances. This repository must stay in a consistent, coherent state. OpenCog instances may use this repository to communicate with one another. Lets call this the Repository.

III) There is a device-driver-like software shim between AtomTable and the Repository. Lets call this the driver.

IV) There is an API between AtomTable and driver. Lets call this the backing store API.

V) There is an API between AtomTable and MindAgents, etc. Lets call this the Atom Table API.

Details

II.a) The repository is a server. It might be, but need not be, another opencog instance. It might be a traditional "database". It may be bigtable-like, e.g. based on HyperTable of HBase. It should not be a single-user disk volume.

II.b) A single-user, local-disk-volume storage backend could/would be useful, but I don't yet understand how this should layer hierarchically with the shared-server of II.a above.

II.c) repository provides some amount of "smarts" for queries, e.g. possibly map-reduce-like. Just how much smarts is open for debate. I've been assuming a small/moderate amount of smarts, i.e. enough to keep a few indexes, maybe perform a few basic joins, but this may be a bad assumption, and we should consider a repository capable of arbitrarily complex queries.

II.d) Cost of accessing repository is "high". Thus, although the repository may maintain indexes of various sorts, the cost of accessing those indexes is also high.

II.e) The repository need not be a "single" server; it may be P2P-like or have whatever architecture is needed to provide scalability so as to allow for simultaneous use by millions of opencog instances.

III.a) The driver might contain indexes; these indexes are "by definition" in RAM, and are low-cost to query.

I.a) The AtomTable already contains 6 different indexes.

I.b) There already exists a "completely general" query mechanism for obtaining atoms that are in the AtomTable.

Issues

VI) What's the backing-store API?

VII) Should the AtomTable API change? How?

VII.a) What kind of queries are supported by the AtomTable API?

VII.b) What sort of indexes should AtomTable keep?

VII.c) Should there be user-definable indexes kept by the AtomTable?

VII.d) What's the API for these user-definable indexes?

VII.e) Should there be locking primitives in the API?

VI.a--VI.e) same as above, but for the BackingStore API.

VIII) Should there be locks in the repository? What would their semantics be?

IX) If the repository has been updated by some other clients, do do these changes need to be communicated to the local AtomTable? Does the local AtomTable need to be kept in sync with repository? If so, why? How stringently? Can this be avoided? (I think the answer to the first question is "no", if so, this is a requirement, not an issue.)

X) Should attention values be stored in the repository? The answer seems to be "no" if the repository is used only for storage for currently-running MindAgents. However, the answer would need to be "yes" if there is a desire to take a snapshot of a running MindAgent, and restart the snapshot later.

Requirements

CI) A change-notification system is needed. OpenCog instances must be able to receive events from the repository, indicating atom creation, request-for-deletion, changes in importance, etc. Events may be generated from the repository itself, or by other opencog instances.

CI.a) These events should be configurable, so as not generate a flood of events delivered to one opencog instance as the total number of instances connected to the repository grows.

CI.b) opencog instances may use other communications channels, as desired; the above channel is intended for repository-related messages.

CI.c) Is there any reason to restrict the kind of use that this event channel is put to?

CII) Just because an Atom has been created within the AtomTable, does not mean that it must be pushed out to the repository. i.e. there is no requirement to keep the repository strictly in sync with the local AtomTable.

Some Further Comments From Joel

Forgetting versus Dropping

Ben mentioned "forgetting" and then talked about either removing the atom completely or pushing it out of the local AtomSpace and into the Repository. These are distinct problems. I propose "forgetting" is only used for complete removal from the Repository (although CogNodes may still have local version). I propose that "dropping" an atom from a CogNode, means flushing it from the in-memory cache, whether or not it has been previously stored permanently in the Repository.

Obsolescence versus Deletion

BigTable is actually a persistent data structure. Thus it has copies of all the older versions (and the length of time the previous versions are stored can be tuned) and one possibility would be to mark atoms as obsolete when they are deleted, but all the previous version would still exist. New links to the latest version of the atom could no longer be made, but instances with references to earlier versions of the handle could still exist... if the instance tries to store a new link (that refers to the old version of the atom) then it'd have to decide whether to recreate the atom, or forget the link, when "dropping" them to the Repository. Perhaps a periodic down-time or "sleeping" phase would drop all local references and allow the Repository to flushed of removed versions (and for that matter, other earlier versions of atoms that are not current too).

Should Handles Have Timestamps?

The above bit about versions in BigTable also indicates that Handles could have a version or timestamp. Beyond supporting the above, this would allow more intelligent revision of TruthValues when different CogNode instances drop their local versions to the Repository.

On the other hand, the current Atom/Handle implementation is already quite 'bloated', using a considerable amount of RAM for every atom. Are timestamps really worth the added storage cost?

Attention Allocation Heuristics

Attention allocation isn't just for memory management and forgetting/dropping atoms.

  • high STI and low LTI atoms should probably not be stored to the Repository unless/until their LTI increase.
  • low STI atoms should probably be dropped from local instances.
  • old and very low LTI atoms should probably be forgotten from the Repository.
  • since STI only really makes sense in terms of in-memory atoms, perhaps the Repository shouldn't store this information.
  • Does it make sense to store LTI in the Repository?
  • Atoms freshly retrieved from the Repository would be given 0 STI. Huh? Why? Because retrieval is expensive, some mind-agent must have decided it was worth the effort to retreive them. This implies that they would have a high STI, else, there's no need to retreive!
  • VLTI (very long term importance) should immediately be put in the Repository.

These are basic guides, but more intelligent forgetting and dropping of atoms will eventually be implemented (given two low LTI atoms, the one that is hardest to recreate, or has more knowledge value, should be forgotten after the other).

(Comment from Ben.) Or if there are 100 medium-LTI Atoms, but these can be quickly and somewhat accurately generated from 20 of them, it may be best to delete the other 80 .. even though they're nowhere near the lowest-LTI ones in the AtomTable.

Possible Use of SmartPointers

I want to know more about the Boehm GC... but I also think the Boost Smart pointer system would work well:

use weak_ptr<Atom*> for all atom references, except for the AtomTable map which would have a shared_ptr<Atom*>.

weak_ptr implements locking semantics to guarantee the pointer doesn't get freed while it's being used (see http://www.boost.org/doc/libs/1_38_0/libs/smart_ptr/weak_ptr.htm)

Levels of Accessibility

Imagine we have a node for the general concept of "cat" in RAM, and there are

  • nodes for 50 specific cats stored in RAM, and linked to from the "cat" node
  • links from the "cat" node to 75 additional "specific cat" nodes stored on disk [these are links that point to Handles corresponding to nodes currently living in disk but not RAM] ... I'll call these 75 "semi-accessible" nodes
  • nodes for 5000 other specific cats stored on disk

If PLN needs to do some reasoning about cats, in many cases just following the links from "cat" to the 50 specific cats in RAM should be good enough.

If it needs to dip into the other 5000 cats, there's nothing simple for it to do, except just load some randomly, and see where that leads it. It may want to use some heuristic like loading the ones that used to have the highest average STI values or something...

On the other hand, if it wants to dip into the 75 "semi-accessible" specific-cat nodes on disk, then the MindAgent has some other alternatives available. Maybe it knows it's specifically interested in American cats, and it can then select the 25 out of these 75 semi-accessible on-disk specific-cats that are also known to be American (based on the links in RAM), and just load those...

...

Note I have referred only to disk but not the network here; but the same basic logic holds if you replace the 75 nodes on disk with 75 nodes on remote machines, or a mix of nodes on the local machine with nodes on remote machines, etc.

More Detailed Follow-Up Suggestions

From Joel...

Also note that the Request objects I was talking about are meant to encapsulate these situations. I propose assigning a cost to each storage (say 0 for memory, 1 for disk, 2 for network). Then a caller can refer to the maximum depth of search, but also alter the depth if more results are required. The request object would return only the 100 or so most important results to begin with, but the caller could request more if that's not sufficient, essentially splitting the results into chunks.

What follows is an example of how I see requests occurring: 


#define R_MEMORY 0
#define R_DISK 1
#define R_DISTRIBUTED 2
Request::Request(Type t, int maxDepth=0, int buffer=100);

Then, to get all CONCEPT_NODEs (pretty much guaranteed to be a huge result)

Request r(CONCEPT_NODE, R_DISK);
// On the first fetch, the in memory handles are retrieved first, even if
// the the maxDepth is set for disk.
r.fetch();
bool enough = false;
foreach (result, r, true)
// true indicates that the Request object should automatically request
more results
// once the buffer is empty...
{
   // various operations, if enough results processed then break out of loop.
   if (enough) break;
}
if (!enough) {
   // Going through all memory and disk handles was not enough...
   // alter the request to also check the distributed AtomSpace.

   // distributed requests are expensive, so limit to 2.
   int maxDistributedRequests = 2;
   r.setMaxDepth(R_DISTRIBUTED);
   while (maxDistributedRequests > 0 && !r.isComplete())
       foreach (result, r, false) {
           // Operations on distributed handles, bearing in mind these are
           // likely not to be in memory.
      }
   }
}

I think a similar interface for incoming sets could also work.

Results should be ordered by STI for in memory atoms, and is random (or at least has no guaranteed order) for disk and distributed.