FileStorageNode

From OpenCog

The FileStorageNode is a StorageNode that can save/restore Atoms to a plain flat file. It offers a basic subset of the StorageNode API, suitable for reading/writing files, but without the ability to search and query.

The FileStorageNode is intended for high-speed, compact backup and restore of AtomSpaces. It is faster, and more compact (after compression) than the RocksStorageNode. This is because the flat-file format does not contain any indexes. It is also 10x faster than using scheme (guile) to dump Atoms by hand. This is because the flat-file uses optimized read/write code.

After compression, typical Atom sizes are 4 to 10 bytes. Thus, even large AtomSpaces can have a rather modest disk footprint, when stored as compressed s-expressions.

Example

Here's a very short example that opens a file, and dumps the entire AtomSpace:

(define fsn (FileStorageNode "/tmp/foo.scm"))
(cog-open fsn)
(store-atomspace fsn)
(cog-close fsn)

Reading the file is equally straight-forward:

(define fsn (FileStorageNode "/tmp/foo.scm"))
(cog-open fsn)
(load-atomspace fsn)
(cog-close fsn)

Individual Atoms and Values can also be written; see the section below. Complete, functional demos are provided in the github examples directory. See in particular persist-store.scm.

URL format

The URL format simply specifies a location in the file system:

file:///some/path/to/file.scm
/some/path/to/file.scm

Note that the first form has three slashes in it, not two. The filename must be an absolute path in the filesystem, i.e. it must start with a slash.

The file will be created, if it does not exist. It will be appended to, if it does exist.

Usage Notes

The FileStorageNode implements only a subset of the full StorageNode API. It implements load-atomspace, store-atomspace, store-atom, store-value and barrier.

Thus, besides loading and storing entire AtomSpaces, one can selectively store individual Atoms. The store-atom call will append one Atom, and all of it's Values, to the file. The store-value call will append one Atom, and only the one indicated Value to the file. To save only some of the Values on an Atom, just call store-value on the selected Values. The same Atom can be specified multiple times within a file; during loading, the set of keys on the Atom will be a union of the ones in the file, with the last version of a Value on a given key being the one that will be loaded into the AtomSpace.

The barrier call forces a file write. Writing is buffered (by the operating system). A write might not appear in a file until after the barrier call returns, or after sql-close returns.

Implementation

The source code for this node is located here: https://github.com/opencog/atomspace/tree/master/opencog/persist/sexpr

See also