|
NEP technical guide |
|
|
Thomas Krichel will, over the next few years, set up NEP_Watch, a comprehensive system that will allow to monitor what editors are doing, and who NEP as a whole performs. As part of this effort, you already have
There is a set of steps to go through to find a set of new papers that have come out. This has to be done once a week, say on a Sunday.
nohup make_all.pl >
~/make_all.log &cdtail -f make_all.logtail make_all.logapache_restart
Forcing reload of web server: Apache2 ... no pidfile found! not running?
(98)Address already in use: make_sock: could not bind to address 0.0.0.0:1998 no listening sockets available, shutting down
Unable to open logs
apache_fix
as well as execute
apache_restart
again.
nohup presorting.pl > presorting.log
Two other features of this process should not go unnoticed.
make_all.pl. For example
nohup make_all.pl 2005-06-05
> ~/make_all.log &presorting.pl. For example
nohup presorting.pl
2005-06-05 > presorting.log &presorting.pl will exit with an error. Otherwise
it will presort the nep-all issue for the given
date. If you give no explicit date, ernad will
search for the last date of a nep-all issue
and presort that one.
Until August 2006, the gathering of report histories was a very
cumbersome process. There have been several kinds of log files parsed
but all are mail files. They are therefore highly unreliable, for the
pre-ernad period especially. All parsing was done by the script
~/perl/nepmailparse and the resulting AMF data stored in
~/public_html/amf/nep/, one file per report. These files
are still there. They are used for the gathering of pre-ernad
historical data, but they are no longer being updated.
The new historic file system is known as repsum. The main innovation that they introduce is that they credit individual editors for each issue. To aim is to get a complete history of all editors. This can be used to customize the experience length.
There are three scripts to maintain repsum data.
~/perl/repsum_mail parses the data from the email-based
record files, in the old directory
~/public_html/amf/nep/. It adds historic editor
information as it is maintained in the historic editor file
~/var/pre_ernad_editor_data.txt.
~/perl/repsum_mail only needs to be run when the date in
the historic editor file is changed. How to do that is described in
the next section. The ~/perl/repsum_mail generates some
Perl output in ~/var/repsum/mail. This Perl output serves
as input for repsum_all.
~/perl/repsum_ernad parses the sent
directory for every report,
~/ernad/var/reports/report/sent
directory for every report report for historic records. To
find the editors that are associated with report issues, it is
sufficient to parse the report issue file (rif). If that information
is not found in the rif, the script applies some backward heuristics.
These heuristics relates most of this issues to identified
(i.e. RAS-registered) editors. Those that are not resolved are
reported upon by the script. The script generates some Perl output in
~/var/repsum/ernad. This Perl output serves as input for
~/perl/repsum_all.
~/perl/repsum_all parses the files in
~/var/repsum/ernad and ~/var/repsum/mail. It
extracts the person's short-id out of the homepage date, and splices
the mail and ernad data together, adding mail data only when ernad
data is not available. The script generates some AMF XML output in
~/public_html/repsum/. The output is self-explanatory,
and mirrored to the conf directory of RePEc:wop.
It remains that one advantage of mail file parsing is that it gathers data when the recipients of the report have received it, rather than at the time when the report created. Thus current NEP historical records will contain data from reports that have never been distributed via email.
This simple text file should contains information about
editors in the pre-ernad area, basically from April 1998 to
January 2005. Lines that start with # are
ignored. Other lines should have the form
report id start end name
where report is the report,
id is the short-id of the editor,
start is the start date,
end is the end date, and name
is the name of the editor. Here is an example
nep-ecm pka1 1995-01-01 2006-01-01 Sune Karlsson
You can have more than one line per report, but please make
sure that the editorial time periods don't overlap.
When a change has been made to the file, a
you have to run
~/perl/repsum_mail to make use of the changes
in the data that is gathered. Note that changes to the
file only affect the pre-ernad report history.
When an editor finds no relevant paper in a nep-all report,
she can choose to delete the nep-all issue.
When an editor pushes the
delete button, the report is not deleted.
Instead, an
empty report is written into
/home/ernad/var/reports/nep-xxx/sent.
Here nep-xxx is the report id. Thus, if
yyyy-mm-dd is the issue date, you just have to do
ernad@snefru:~$ cd ~/ernad/var/reports/nep-xxx
ernad@snefru:~$~/ernad/var/reports/nep-xxx rm sent/yyyy-mm-dd*
However, if the editor has also already gone through the creation of the report, but wants to reissue the report because, for example, there has been a problem with the distribution of the report, all files for the report issues will have to be delete. In that case, there is some more stuff to delete
ernad@snefru:~$ cd ~/ernad/var/reports/nep-xxx
ernad@snefru:~$~/ernad/var/reports/nep-xxx rm created/yyyy-mm-dd*
ernad@snefru:~$~/ernad/var/reports/nep-xxx rm selected/yyyy-mm-dd*
ernad@snefru:~$~/ernad/var/reports/nep-xxx rm ordered/yyyy-mm-dd*
ernad@snefru:~$~/ernad/var/reports/nep-xxx rm sent/yyyy-mm-dd*
Leave the files in the source and svm_light
alone.
The command is
~/mailman/bin/change_pw -l LIST -p NEW_PASSWORD
where LIST is name of the list and
where NEW_PASSWORD is the new password.
All information about reports is kept in
~/ernad/etc/reports.xml. Before you do
that, make a security copy of the file. If the current date
is yyyy-mm-dd, do
ernad@snefru:~$ cd ~/ernad/etc
ernad@snefru:~/ernad/etc$ cp reports.amf.xml reports.amf.xml_yyyy-mm-dd.amf.xml
to keep a copy of the file before you potentially screw thing up. The file documents each list in blocks such as
<haspart>
<collection id="RePEc code of nep report">
<title>subject</title>
<homepage>http://lists.repec.org/mailman/listinfo/NEP_report_code</homepage>
<ernad:password>password of editor for ernad</ernad:password>
<haseditor>
<person>
<name>name of editor<name>
<name xml:lang="en">English transliteration of editor's name<name>
<homepage>homepage of editor</homepage>
<email>email address of editor</email>
<ispartof>
<organization>
<name>name of institution</name>
<homepage>homepage of institution></homepage>
<postal>country name only</postal>
</organization>
</ispartof>
</person>
</haseditor>
</collection>
</haspart>
The above should be self-explanatory.
You have to save the file in UTF-8 encoding when changing it. One way to do this is to use the emacs editor directly on snefru.
When you have made your changes, check the file with
ernad@snefru:~/ernad/etc$ xmlwf reports.amf.xml
It this prints out anything, copy the saved version back immediately. If it does not complain, install the documentation with the command
ernad@snefru:~$ ~/perl/makedoc_nep
Note that this will also update the subscriber numbers to the latest found in the data. There should be data from the first of the current month. If this is not there, notify Thomas Krichel immediately.
Finally, execute
apache_restart
No person must be allowed to work as an editor unless that person is registered with the RePEc Author Service. At the time that the personal information is added the RAS pages of the person must be added as well. If they are not there, the editor looses credit for editing and may also loose data fed into her pre-sorting information.
When opening a new report, say nep-NEW, you should first agree with the editor the email address EMAIL_OF_NEW_EDITOR that she will be using and the Mailman password PASSWORD_AGREED_WITH_NEW_EDITOR that she will be using. The general password for NEP should not be used as a password for individual lists.
Once you have the email of the editor, subscribe the address to the nep-editors list. We need to check periodically that all editors are on that list.
Next, carefully enter the report information into
reports.amf.xml as detailed in
the previous section. Then
you have to run the ernad command that makes ernad
recognize any new report
cd
cd ernad
cd perl
./processreports.pl
Then, execute
apache_restart
to make the new reports.amf.xml file visible
in ernad.
The next step is to set up the list. To do this, become the user "list":
ernad@snefru:~$ su list
Password:
Type the password for the user list here. Then create the list with the newlist command.
sh-2.05b$ /var/lib/mailman/bin/newlist nep-NEW
Enter the email of the person running the list: EMAIL_OF_NEW_EDITOR
Initial nep-NEW password: PASSWORD_AGREED_WITH_NEW_EDITOR
It will then say
To finish creating your mailing list, you must edit your /etc/aliases (or
equivalent) file by adding the following lines, and possibly running the
`newaliases' program:
## nep-NEW mailing list
nep-NEW: "|/var/lib/mailman/mail/mailman post nep-NEW"
nep-NEW-admin: "|/var/lib/mailman/mail/mailman admin nep-NEW"
nep-NEW-bounces: "|/var/lib/mailman/mail/mailman bounces nep-NEW"
nep-NEW-confirm: "|/var/lib/mailman/mail/mailman confirm nep-NEW"
nep-NEW-join: "|/var/lib/mailman/mail/mailman join nep-NEW"
nep-NEW-leave: "|/var/lib/mailman/mail/mailman leave nep-NEW"
nep-NEW-owner: "|/var/lib/mailman/mail/mailman owner nep-NEW"
nep-NEW-request: "|/var/lib/mailman/mail/mailman request nep-NEW"
nep-NEW-subscribe: "|/var/lib/mailman/mail/mailman subscribe nep-NEW"
nep-NEW-unsubscribe: "|/var/lib/mailman/mail/mailman unsubscribe nep-NEW"
Hit enter to notify nep-NEW owner...
You can ignore the requests to create aliases, they are set up in Exim, so you don't have to worry about this. So hit enter there, and then exit leave the user list
sh-2.05b$ exit
exit
ernad@snefru:~$
After the list has been opened, it must be configured. Refer to the next section on how to do that.
This is done with the special script "configureneplist".
ernad@snefru:~$ configureneplist nep-NEW
reconfiguring list: nep-NEW
/home/ernad/mailman/bin/config_list -i /tmp/nep-NEW.conf nep-NEW
Please note that script is not dumb, it will check first if there is
list line for nep-NEW in ~/ernad/etc/reports.amf.xml. If this
script reports another mistake, inform
Thomas Krichel immediately.
Finally, you must check ernad@lists.repec.org to
the list members. The list configuration should do this
for you, but double-check, because if this guy is not
on the list we do not have proper logs.
This is useful when you want to recreate a list afresh to hopefully cure a problem.
ernad@snefru:~$ /var/lib/mailman/bin/list_members list
admnep@list.repec.org
admnep@nep.repec.org
bparks@wueconc.wustl.edu
christian.calmes@uqo.ca
ernad@lists.repec.org
ernad@nep.repec.org
ernad@openlib.org
generaleditor.nep@uwe.ac.uk
gilles.sanou@uemoa.int
jeffrey.robertson@aph.gov.au
library@ote.gr
malarselvam@rediffmail.com
marco@novarese.org
thivant@univ-lyon3.fr
walter.frisch@univie.ac.at
You can save the list in a file for later use
ernad@snefru:~$ /var/lib/mailman/bin/list_members list
> file
where file is the name of the file you want to send the list of members in. When you later add the members back to the re-created list, please note that you should switch off the sending of the welcome message. By default it is switched on. If the members get that message, they will be very confused.
Become the user list
ernad@snefru:~$ su list
Password:
Remove the list
sh-3.00$ /var/lib/mailman/bin/rmlist list
Not removing archives. Reinvoke with -a to remove them.
Removing list info
Then leave the user list.
sh-3.00$ exit
exit
