Fort Lee paper
This is the Fort Lee paper. It sets out the an initial specification
for a dark archive for RePEc full-text and templates.
The first draft was of 2012‒11‒07. This is the version of 2014‒03‒12.
The goal is to provide a framework of the storage of persistent
copies of the RePEc digital library. For the purpose of this
document, the RePEc digital library consists of template metadata for
documents, and of full-text copies of these documents referenced in
metadata records, if such full text can be downloaded from a non-priviledged
computer. Thus full text can be assumed to be publicly available.
- object of interest (“obi”)
- This is anything that has a URL or a RePEc handle, and its URL or RePEc handle is found in RePEc data or in ArchEc data.
This is the set of all RePEc template metadata known now.
- ArchEc data
ArchEc data is RePEc template data stored by ArchEc at a previous
time, or objects of interest stored by a run of ArchEc to gather file
objects of interest.
- template object of interest (“tobi”)
- a template found in the RePEc data, as a string. Its handle is the RePEc handle.
file object of interest (“fobi”)
something that is pointed to by a file-* construct in RePEc's ReDIF
date. Its handle is the URI, subject to normalization of URIs. (FIXME: say
what this is.)
- obi store
- a way to store the obis. The obi is
stored on disk. This is described below. For the moment, let’s just
assume that the obi store exists, and it creates a permanent disk
location of the obi.
- unix time stamp at a time
- software tist
- The tist at which we start to make a gathering run of ArchEc.
- tobi tist
the tist of a file in the obi store which a tobi is found.
When tobi store software is run, either an
- tist found
The tist of the time we find an obi for the first time. This is
either the tobi time or the software tist of a run where we first
get a copy of the fobi.
- tist noted
- the software tist we find an object for the
first time, or any time we notice that its contents has changed from
the last time we observed it.
- tist lost
- the software time we find that an obi that we
have observed is no longer there. Such an obi is called lost.
- this is a pointer to file information we find in
the ReDIF data
- either appearance, change, or deletion of an obi.
The object store depends on time. Therefore time handling is quite important.
RePEc is externally furnished to ArchEc and periodically updated. This is the
parametere ReDIF_directory in the configuration file.
For the purposes of building the a gathering application static tarballs of old
RePEc data are also available. The tarball location may be supplied
with the -f argument at start time. In that case the ReDIF_directory
parameter in the configuration file is ignored.
The obi start location is specfied with the
object_store parameter in the configuration file.
the object store is in "archive/", sources are in "data/". That
directory has "tarballs/" storing past tarballs, by data, and
"current/" linking to the current dataset.
We use the tist noted. Invert it by chars, set a / after the
third char and the sixth char for the first occurrences of the tist.
Example. Store the homepage of Thomas Krichel http://openlib.org/home/krichel
at some observed tist.
krichel@trabbi:~$ date +%s
invert the string to get 0321845231 and split for directories 032/184/5231
Take URL and uri_escape_utf8 (in Perl)
The file is the store is
but since there is a redirect to http://openlib.org/home/krichel/,
it stores at
If there is a redirect from URL foo to URL bar, make foo a symlink
to bar. Exception: if foo and bar differ by a trailing slash, omit
For the object store, we need a caching database. We use
MongoDB for this purpose. Not the caching database just
for caching. It is used to hold any data that could not
be derived again.
We have four collection.
Of these four, the two in the second column are not implemented.
The data collections, “m”, link an id of an obi to a hash. The hash
matches tists to md5 sums. The file can be trivially constructed from
tist and obi id.
The checksum collections, “v”, links an md5 checksum of the text in the
file—rather than of the file—to a location in the the obi store
where the checksumed contents is held.