Description

Usenet-Web 1.0.2 is a combination Usenet newsgroup archiver and archive Web presentation system. It is not intended as a substitute for a newsreader, and so does not include any facilities for posting to the Usenet. It does include facilities for searching the archive 'From' and 'Subject' lines for ranges of dates and can handle multiple newsgroup archives.

Why would I want to put a Usenet group archive on the WWW?

Several possibilities come to mind:

To make a local newsgroup (such as mysystem.announce) available off site or as a service to on site people.
To retain a permanent record of a newsgroup in a usable, searchable, format. Many existing archives are completely useless because they are impossible to effectively find things in.
Because you want to. :-)

What do I have to do?

I have attempted to make installation, configuration and maintenance as simple as possible - but you will still need some knowledge of how your system is configured to make it work correctly.

Usenet-Web 1.0.2 does not contain any direct support for NNTP - but it is still possible to use it if you do not have a local news spool, but do have NNTP access. Check under Advanced Topics for details.

If you want to see a Usenet-Web 1.0.2 (actually, 1.0.3 development - the main difference is that 1.0.3 development now supports mail lists) archive in operation, try the one maintained at http://www.netimages.com/ni-cgi-bin/fetch.

Usenet-Web 1.0.2 is compatible with Nathan Neulinger's cgiwrap program.

System Requirements

Perl 4 or later.
A local Usenet news spool - NNTP is not directly supported at this time.
GNU 'gzip' and 'gunzip' or 'zcat'
Sufficient storage to store the archive.
Ability to install and run CGI scripts.

Return to Index

The distribution for Usenet-Web 1.0.2 is a gzipped tar file named usenet-web-102.tar.gz. This file contains the following source and resource files:

fetch (32.5K): CGI script for presentation of a Usenet-Web archive on the World Wide Web.
usenet-web-archiver.pl (6.6K): The software for archiving newsgroups. Multiple news groups may be archived by a single invokation. Ideally it should be called regularly by a cron job.
usenet-web-rc.pl (3.5K): Routines for reading and manipulating the Usenet-Web resource file. These routines are required by all the other perl programs in the distribution.
usenet-web.rc (3.5K): The Usenet-Web resource file. This contains most of the configuration information needed by Usenet-Web. This is an example file to start you.
usenet-web-index-rebuild.pl (9.5K): This program rebuilds the index files used by Usenet-Web. It is intended to allow recovery from corrupted index files. It optimizes the index files for the fetch program's use and will also convert index files for Usenet-Web 0.9 to the form required by 1.0.2. For safety - it renames the original master index files with the extension '.old'.
IMPORTANT: It would be a very Bad Thing [tm] to run the usenet-web-index-rebuild.pl program at the same time as the usenet-web-archiver.pl. Make SURE that any cron jobs that could run the usenet-web-archiver.pl program are disabled before you run usenet-web-index-rebuild.pl.
graphics/icons/: This directory contains all the graphics/icons used by Usenet-Web.
cache/: This is an empty directory needed by the fetch program for proper operation. It is included to simplify configuration.

Return to Index

Obtain the distribution file usenet-web-102.tar.gz and place it in the directory you wish to install the package in. I recommend against this being in a cgi-bin directory. You should install to another directory and then copy only the 'fetch' program to the cgi-bin directory along with a symbolic link to the 'usenet-web-rc.pl' program. This will be explained in detail later in this document.
Extract the distribution files by typing
- gunzip usenet-web-102.tar.gz
- tar -vxf usenet-web-102.tar
from the Unix shell prompt.

You should now have a directory named 'usenet-web-102', containing the distribution files.

Configuration consists of two steps:

Determining your system setup
Modifying Usenet-Web as necessary to match your system setup.

You will need four important pieces of information before you start:

The location of your news 'active' file: This is the file used by your news system to track active Usenet newsgroups and what messages are currently stored in the news spool. It can be in any of several places, some possibilities are '/usr/spool/news/active', '/usr/local/lib/inn/active' and '/usr/lib/inn/active'. If you do not know where this file is stored - try asking your system administrator.
If you absolutely cannot find it - you will have to configure Usenet-Web to operate without it. Check under Advanced Topics.
The location of your news spool: This is the location where all news articles are stored normally. Most frequently this is '/usr/spool/news'
The location of 'gzip' and 'zcat' (or 'gunzip'): Usenet-Web stores articles compressed to save room. You should be able to locate where 'gzip' and 'zcat' are stored by typing 'which gzip;which zcat' at the Unix shell prompt. If you cannot locate 'zcat', try 'which gunzip' instead.
The location where you intend to store the archived articles: Usenet-Web will by default create a newsspool-like directory structure to store your archived files in. The directory you choose as the root of this directory tree must be writable by yourself and readable by all. This means file access privileges for it should be set to 0755.

First, move all the graphics/icons in the 'icons/' directory to whatever directory you store graphics/icons in.

Now edit the 'usenet-web.rc' file to match your site's configuration and the newsgroups you desire to archive. Questions about the 'usenet-web.rc' file can probably be answered by looking under Advanced Topics.

Next, edit the line in 'usenet-web-rc.pl' that reads

$RESOURCEFILE="/home/users/s/snowhare/usenet-web/usenet-web.rc";

to point to the actual location of the 'usenet-web.rc' file.

Move the 'fetch' program to the CGI script directory and make a symbolic link in the CGI script directory pointing to the 'usenet-web-rc.pl' file.

Now you are ready to cook.

To get the ball rolling (and assuming you have correctly configured everything), run the 'usenet-web-archiver.pl' program. This could take a few minutes as the program scans all current articles and adds them to the archive. If everything works correctly, you should get no error messages. If you do get an error message: read it! I have tried to make them informative enough to guide your towards the problem. If that doesn't work - check under Troubleshooting.

Your archive should be intialized and ready for use. To check it - open the URL that points at the 'fetch' script. For example, if you have installed 'fetch' in the system's main cgi-bin directory, you would use <URL:http://my.domain.address/cgi-bin/fetch> (replacing 'my.domain.address' with the name of your server).

If it works - great! You are now the proud owner of a spanking new Usenet-Web 1.0.2 archive. Time to make a cron job that runs the archiver on a regular basis.

If it doesn't work, check your configuration again and then try the tips in the Troubleshooting section.

Return to Index

Possible Problems

Using Usenet-Web 1.0 or 1.0.1 rather than 1.0.2

If you were one of the unlucky first sixteen people to download Usenet-Web 1.0 you have two serious bugs in the 'usenet-web-archiver.pl' program. The symptoms are 'Invalid Resource' errors and directories and files may be created without world readable permissions.

If you are using Usenet-Web 1.0.1 or 1.0, there is a bug in the 'usenet-web-index-rebuild.pl' program that can corrupt the indexes if you have more than one newsgroup being archived when you rebuild the index files.

Both of these problems are fixed in the 1.0.2 version. Download the 1.0.2 version and replace the 'usenet-web-archiver.pl' and the 'usenet-web-index-rebuild.pl' programs with the new versions.

Incorrect file permissions

Double check that your file permissions are set correctly on all the files and directories needed by Usenet-Web. In case of doubt, try setting the permissions to 0777 and see if that affects the problem. Then fine tune the permissions for security.

usenet-web.rc misconfigured

Make sure that all the paths and directories are specified correctly. For the ICONPATH and URLPATH - remember that they use URLs - not unix directory paths.

$RESOURCEFILE not set correctly in 'usenet-web-rc.pl'

This is a choke point for all the other programs - if 'usenet-web-rc.pl' can't find the 'usenet-web.rc' file, nothing else will work.

No symbolic link to 'usenet-web-rc.pl'

You must have a link to 'usenet-web-rc.pl in the same CGI directory as the 'fetch' script or it will not be able to function.

Return to Index

Benjamin "Snowhare" Franz /
snowhare@netimages.com