Object Tracking Buffer

From OpenCog
(Redirected from Visual Perception Buffer)
Jump to: navigation, search


Caution! This page describes a subsystem that seems to work, but is currently unmaintained. It's also more-or-less deprecated, for the reasons given below. See the SpaceServer page for a description of how it should be implemented, and how it would be used. The code is located here: opencog/spacetime.

Object Tracking Buffer

This page describes a system for tracking the spatial positions of objects in opencog. It has several fairly deep design/architectural flaws.

  • It is non-modular, centralized, monolithic, non-scalable. This is because it got built on an octree (but would be no better if it was a point-cloud). There is no need to store space-time values in a single, centralized location; using Values, as proposed on the SpaceServer page, can achieve the same utility, and more, in a decentralized, modular fashion.
  • It is missing pretty much all API's needed to be useful: no particular way to query, interrogate the relative positions, sizes, distances between objects, or their movements.
  • There is no way to modularly add new abilities to the server (e.g. for size, speed) without turning the server into a big complex does-everything-for-you server.

For these reasons, this code is deprecated for future use! On the other hand, it "mostly just works" for what it does, and thus one can get away with "just using it", for now.

Theory

This page outlines a design for an OpenCog "visual perception buffer", which will contain the last N moments of perception coming into a system. There is some architectural complexity here in that the same OpenCog system may perceive multiple different regions or game worlds concurrently, and within each world may use multiple sensors.... Also there are decisions to be made in that not everything passing through the short-term visual perception buffer can make it into long-term memory (due to RAM size limitations) ...

We introduce the notion of an "Embodiment" -- meaning a bundle of sensors that are intended to be considered together, and that observe aspects of a common world. In terms of the Timeserver, all the sensors in the same embodiment can be assumed to refer to the same TimeDomain.

The basic idea is that each Embodiment has, optionally, a PerceptualBuffer associated with it.

The PerceptualBuffer contains a circular queue of Octomap arrays ( some explanation of Octomap here), where the k'th OctoMap array refers to a point in time k time-units before the present (and the length of the time-unit is a configurable parameter). Each Octomap array contains a set of Octomaps, corresponding to different coordinate systems within the same Embodiment... For simplicity we can now assume that, for a given Embodiment [e.g. a specific Hanson robot, or a specific game of Minecraft, etc.], there is a fixed set of coordinate systems that matters.

In general, the number of coordinate systems used in a certain PerceptualBuffer should be the minimum required. If there are multiple fixed-position sensors observing an overlapping scene, and it's possible to do some math to map their outputs into the same OctoMap, then this may as well be done in preprocessing. OTOH if your robot has a camera on its hand then sometimes you may not be able to align the output of the hand camera with the output of the eye camera... so you need 2 OctoMaps in the array in this case, one for the eye camera and one for the hand camera..."

A novelty/salience detector is used to identify notable objects in an OctoMap (e.g. large objects or faces that move, or that appear in the scene for the first time...). When an object is noted as novel/salient its STI is boosted to a higher level. (By default each observed object gets a nonzero but low STI...)

(The very first version of novelty/salience detection can just detect if a new object has appeared, or if something has moved. But to do this well we need to use some information theory to measure "visual surprisingness"... this has been done before...)

When the last Octomap-array in the queue is about to get cleaned out (to make room for the new objects observed in the present moment), any Atom it points to that has a reasonably high STI is supplied with atTimeLinks and atLocation links indicating its time and location. Atoms that it points to whose STI is too low, can have their STI set to 0 and get removed.

Note: when an Atom is removed from Atomspace, any pointers to it must be removed from any Octomap that points to it... it must also be removed from TimeServer, etc.

However, the case of an Atom getting removed from Atomspace by the Forgetting agent, while it is still pointed to by one of the Octomaps in a PerceptualBuffer, should rarely occur in practice. It should occur only if parameters are badly tuned, it would seem. By setting the initial default STI of every perception (even if not novel) and the STI decay rate appropriately, one can have a situation where even "relatively unimportant" new percepts' STI values are not low enough to get forgotten by the time the Octomaps pointing to them are cycled out of the circular buffer.

Note: Simplification of API has occurred and scheme wrapper has been created for managing multiple octomap time buffers. Also basic spatial relations, fixed time unit elapse duration and auto time stepping have been implemented.

VisualPerceptionBuffer.jpg Time-Unit.jpg


Scheme

Functions

These are the functions (as of 2017-02-06):

create-map, get-time-res, get-space-res, get-time-units, step-time-unit, auto-step-time-on, auto-step-time-off, is-auto-step-on, map-ato, get-first-ato, get-last-ato, get-first-locs-ato, get-last-locs-ato, get-at-loc-ato, get-past-loc-ato, get-locs-ato, et-past-locs-ato, get-elapse-list-at-loc-ato, get-elapse-list-ato, remove-location-ato, remove-past-location-ato, remove-curr-ato, remove-past-ato, remove-all-ato

spatial query functions:

get-dist-ato, get-angular-nearness-ato, get-right-left-of-ato, get-above-below-of-ato

C++ API

 
    double get_space_resolution();//map resolution in meters
    duration_c get_time_resolution();
    int get_time_units(){
      return time_circle.capacity(); 
    }
    //current time unit time point and time duration are queried
    bool get_current_time_range(time_pt& time_p, duration_c& duration);
    //helper function to check if a time point is within the Time unit time range
    bool is_time_point_in_range(const time_pt& time_to_check, const time_pt& t, const duration_c& duration)
    {
        return (time_to_check >= t && time_to_check < t + duration);
    }
    //make a new time unit for storage, 
    //should not overlap a previous time unit 
    //and should fall after the previous time unit
    bool step_time_unit();//step_time_unit
    bool is_auto_step_time_on();
    void auto_step_time(bool astep);
    //store an atom at coordinates in map
    bool put_atom_at_current_time(const point3d location,
                              const opencog::Handle& ato);
    bool remove_atom_at_current_time_by_location(const point3d location);
    bool remove_atom_at_time_by_location(time_pt tp,const point3d location);
    void remove_atom_at_current_time(const opencog::Handle& ato);
    void remove_atom_at_time(const time_pt& time_p,const opencog::Handle& ato);
    void remove_atom(const opencog::Handle& ato);
    //get atom at current time unit
    bool get_atom_current_time_at_location(const point3d location,
                            opencog::Handle& ato);
    bool get_atom_at_time_by_location(const time_pt& time_p,
                       const point3d location, opencog::Handle& ato);
    time_list get_times_of_atom_occurence_at_location(
                                               const point3d location,
                                               const opencog::Handle& ato);
    time_list get_times_of_atom_occurence_in_map(const opencog::Handle& ato);
    point3d_list get_locations_of_atom_occurence_now(const opencog::Handle& ato);
    point3d_list get_locations_of_atom_occurence_at_time(const time_pt& time_p,const opencog::Handle& ato);
    //get the first atom observation after a time point
    bool get_oldest_time_elapse_atom_observed(const opencog::Handle& ato,const time_pt& from_d,time_pt& result);//?return location too? 
    //get the last atom observation before a time point
    bool get_last_time_elapse_atom_observed(const opencog::Handle& ato,
                                            const time_pt& till_d,
                                            time_pt& result);//throw
    //AtomList& GetAtomsInLocationBBXatTime();//BBX = bounding box
    //insert point cloud
    //find ray intersection

    //////////spatial-relations-queries
    //should be true-false-unknown
    //assuming z orientation is fixed i.e. sky relative to ground
    //assuming observer is looking towards reference
    //target is $x of reference 
    //y=2-right,1-left,0-aligned,-1-unknown (>elipson,<-elipson)
    //z=2-above,1-below,0-aligned, -1 unknown
    //x=2-ahead,1-behind,0 - aligned, -1 unknown
    point3d get_spatial_relations(const time_pt& time_p,const opencog::Handle& ato_obs,const opencog::Handle& ato_target,const opencog::Handle& ato_ref);
    //not normalized: direction vector -> (target-observer)
    bool get_direction_vector(const time_pt& time_p,const opencog::Handle& ato_obs,const opencog::Handle& ato_target,point3d&);
    //got to another nearness for physical distance, this one is angular
    //2=far,1=near,0=touching, -1 unknown
    int get_angular_nearness(const time_pt& time_p,const opencog::Handle& ato_obs,const opencog::Handle& ato_target,const opencog::Handle& ato_ref);
    //<-elipson=unknown,>=0 distance
    double get_distance_between(const time_pt& time_p,const opencog::Handle& ato_target,const opencog::Handle& ato_ref);
    bool get_a_location(const time_pt& time_p,const opencog::Handle& ato_target,point3d&);
    //constructor
    TimeOctomap(unsigned int num_time_units, double map_res_meters,
                duration_c time_resolution);