Site Network: Prisonsucks.com | Prison Policy Initiative | Prisoners of the Census

Links Modifications

by
Written: Dec 17, 2001

New additions to this page:

We've modified Links 2.0 from Gossamer Threads Inc. to manage the research page. In general, the software is a database that manages links arranged by categories, periodically checks that they are still valid and then updates the links in a variety of formats.

We are making our modifications to the software publicly available subject to the original Links license. You are welcome to use our modifications to the program on your own site to. Unfortunately, the code is now a bit of a mess, so while we can't promise technical support for our modifications we're willing to try and clean up the code a bit if you have a specific question. But be forewarned if you want help modifying it further: We got this to work. But we don't necessarily know why. :)

The modifications are as follows:

  • 1. Puts all my links in 1 list separated by categories with all items sorted by date in each category
  • 2. Print a list of categories along the top so people can jump to that portion of the page
  • 3. Print a whats new list to be included on the front page of the site.
  • 4. Output the 5 most recent links as an RSS 0.91 file, and as javascript includes in a plain format and in a box.
  • 5. Contains additional fields (subtitle, format, organization, month, year, quote, comment).
  • 6. Sorts the links within each category by the month/year assigned to the link, but outputs the month as a world
  • 7. Made a few other mods to make it easier to have multiple setups of links on the same site without confusing them or losing data. (added more names and paths to links.cfg, added admin color variable, added a section variable, etc.) Each installation requires its own directory cgi-bin but it can share some output. I thinking I may want to create similar pages on different topics or countries.
  • 8. Removed jump.cgi and expose URL to the user, likewise remove rating, modify and popularity.
  • 9. Mark categories as "updated" if they are, and links "new" if they are.

A big feature of links 2.0 is that users can suggest links and you must approve them before they show up on the site. Somehow I broke that feature, so it's not included in the mods here.

The modification includes:

Note: the way I have this set up is dependant upon SSI to be consistent with the rest of the site structure. The category list is one file, the link list another, and the whatsnew a third.

The mods can be downloaded as a zip file at http://www.prisonsucks.com/software/linksmod.zip.

The mods contain only the files I changed. You will also need the rest of the distribution from Gossamer Threads.

Throughout all of the files are references to "yoursite.com" that, among many other things will need to be set to your site. The bulk of these are in the links.cfg and templates files, but there might be one or two in the cgi files....

Wishlist:

  • I'd like to get the categories to be sortable. There is a discussion in the links forum for the main category list, but I couldn't get it to work. Note, the categories at the top of the page are generated by the old main page routine. Once we can sort categories there, we also need to work on the tree, which appears to sort differently. Idea: Perhaps I can add a leading character and then trim it with the clean subroutine?
  • Invert sort order for newsletter so newest links are on top whether Ihave many new links. (the forum suggested mods for this didn't work.)
  • Figure out how to make the whats new a list of the 30 newest links with the date prominently displayed. Perhaps have it jump to another page where it gives a chronological history of the links being added? This would be good as sometimes the pages getupdated but there aren't many new links for a while).

Tips:

  • Buy some books on perl, find some web tutorials that you like. (Some are written far better than others.)
  • Search the Links forum. If necessary, enter the routine or variable in question and you'll probably find people talking about it.

New additions to this page:

RSS Module Bug squashed

December 12, 2003

There was a bug where the RSS feed was not validating because I needed to encode the & symbol. There are occasional other problems, such an Amnesty International URL that uses a backslash that I can't seem to encode, but all reasonably frequent URL possibilities should now work in the RSS feed. If you know how to best deal with the backslashes let me know. (The hivelogix url cleaner changed the backslash to a double backslash but I still got an error at feedvalidator.org)

I still can't have the database mark whether I have emailed out a link before, but this matter less now as the new mailing list software can more readily handle more frequent mailings. I'd like to address this in the future, but for now I'm considering this project done. If I don't address this last issue in the next few weeks, I'll repackage the zip file to include all my recent changes.

Search results now balancing tags

December 12, 2003

Balancing the <ul> tags so that search.cgi results would validate was easy. In search.cgi, change

$link_results .= qq|</ul><p>$title_linked</p><ul>\n|;
$link_results .= $link_output{$setoflinks};

to

$link_results .= qq|<p>$title_linked</p><ul>\n|;
$link_results .= qq|$link_output{$setoflinks}</ul>|;

I also had to change some STRONG tags in the script to lowercase as well as make some run of the mill XHTML changes in the templates, but the search results now validate.

Javascript feeds work!

December 12, 2003

I just added an encode_js sub to do the necessary regex and then had to individually send the important variables to it and it works. The javascript output is now more reliable and won't break when I have an apostrophe in the output.

I did a similar thing to get the & symbols to encode elsewhere in the output, as that was breaking the XHTML validation.

Changes to categories

December 9, 2003

I've changed the category listing from being table based to an ordered list. In the css, the list items float with a fixed width, so it looks like a table, but it picks the number of columns that would fit on the screen. During January I'll go through the last of the bugs mentioned on the site and try and prepare a new release. There are a number of issues to clean up in the code so it will validate, and I need to figure out how to encode apostrophes for the javascript feed and & symbols in URLs.

Mailing list updates

October 2, 2003

This is more for my own reference, but I've switched the mailing list from the built in script to pairlist (mailman). I cut the unsubscribe line in admin.pl: (To unsubscribe from this mailing list, just visit: $build_email_url?action=unsubscribe&email=<%Contact Email%>) and made the mailing list the only subscriber. The permissions on email.db where also changed to read-only.

This should give us better verification of new subscriptions and better bounce handling.

search.cgi balancing tags

September 28, 2003

I made some progress on fixing search.cgi so it would balance tags. Adding a close li tag was obvious. Getting it to wrap the li in ul tags is harder. I note that the category results are already working. I was able to hack together something that works, although with an extra close ul tag up front. I need to figure out how i did this in nph-build and whereever it is that the categorie results are built. But at least the results look ok even if it won't validate.

Cleaning up the category listing for css use

September 21, 2003

I made some changes to site_html_templates.pl in the last routine that prints out the category headers as part of my efforts to make the transition to a more css friendly site. I took out the p tags that wrapped the table, but added p tags (open and close) to each cell. I also removed the two nonbreaking spaces on each line. Works much better now.

Avoid category headings from incorrectly displaying as links

September 21, 2003

I changed my style sheets, and found a bug in the tree mod in nph-build.cgi. Where it says $links .= qq~\n</ul><p><strong><a
name="$cat">$title</a></strong><ul>~; it should be instead $links .= qq~\n</ul><p><strong><a
name="$cat"></a>$title</strong><ul>~; (ie. Don't put any text inside the dummy ^lt;a name> tag.)

Mail flags -- less is more.

September 12, 2003

The May 20, 2003 experiment was a success. -f user\@domain.com is enough of a modification necessary to get sendmail to bounce email back to the right address.

Add.cgi not extensible

September 12, 2003

The way I have this now, it's hard to change the variables used because I hard coded the variables I used into sub site_html_add_form in site_html.pl. Why I did this, I'm not sure. I'll see if I can't roll back to the original code from Gossamer-Threads as I believe there it automatically generates the add.cgi form from the links.def information. This might also be the true source fo the problems I wrote about on April 6, 2002 and on December 22, 2001.

Update: Arg. The add.cgi (and the error, email, etc.) stuff is all hard coded in site_html.pl. You have to make the changes there manually. Ouch. Changes also need to be made manually to the add.html files....

Just the flags necessary?

May 20, 2003

I'm experimenting now to see whether just the 

-f user\@domain.com
discussed below is enough.

Bouncing addresses fixed?

November 20, 2002

Ok, I'm not sure exactly which parts of this are necessary, but here is what I did to fix the problem of bounces.

In mailer.pm:
1. In the var declaration change the line to read:

use vars qw($VERSION $error $CRLF @ISA @EXPORT $replyto);

2. Add this line right after the Mailer::Version line:
$Mailer::replyto = 'me@mysite.com'; #added this.


3. On the line before each instance (2) of X-Mailer, add:
print $s "Reply-To: $replyto $CRLF"; #added this

4. Now here is the tricky one, and it might be all you need:
Change each instance (2) of:
-oeq -t ";

to
-oeq -t -f user\@domain.com";

where user@domain.com is the address where bounces should go.

You can not, apparently, set these additional flags in links.cfg as they just get stripped out. My goals are to either fix the regrex or do it with a variable, and in any case find out if I need all these changes. But this is big progress! As far as I know, it fully works now.

Shared templates bug

October 15, 2002

This is an obscure bug affecting very few installations. If you have multiple installations a site, and you set it up to share the templates, the search_error.html template needs to use a variable to represent which page to go to go back and browse the full list. It's currently hardcoded. That link is somewhat necessary as my users think they are using google and searching the links, regardless of what the instructions say. I'll either fix this in the next update, or just tell folks to use the navigation options above....

Minor bugs

October 13, 2002

The apostrophe bug still remains in the javascript output (described below). The mass mailing template in admin_html.pl has some hard-coded text that should be replaced with variables for better mass-installation....

Sending newsletter links only once

September 19, 2002

Is more complicated than I thought. The necessary modifications to the database and sorting routines are easy, figuring out how to tell the database that a link has been sent is more complicated, at least to me. Read below for how far I got and my plea for assistance. Twenty bucks to the first person who figures it out. As always if you have a better way to solve the problem than I suggest, that would be great.

Gossamer Threads Support Forum.

Chronological list feature added

July 25, 2002

We now have a chronological list of links that are too old to be on the whats new list. The list of changes are too numerous to list here, but the main trick was in the build stats routine, to just create another array of all links by date where new = no. We'll repackage the links mods to share this change and all the minor bug fixes soon....

We're almost done fixing the newsletter so we only email out new links once (automatically). Once that's done, we'll release the new package of mods.

Search generates wrong links for categories

July 7, 2002

If you do a search and either a category itself gives a hit, or a category contains a link with a hit, the category URL is incorrect. The fix is in search.cgi. In sub main in the search.cgi change 

  $category_results .= qq|<li>$linked_title\n|;
to 
$category_results .= qq|<li><a href="$sectionhtml#$category">$cat_clean</a>\n|;


Replace the sub build_linked_title procedure in search.cgi with the following:


sub build_linked_title {
# --------------------------------------------------------
# A little different then the one found in nph-build.cgi as it also
# links up the last field as well.

    my ($input) = shift;

    my ($dir, $output, $path, $last, $precleandir);
    foreach $dir ((split m!/!, $input)) {
        $path   .= "/$dir";
        $precleandir = $dir;
        $dir     = &build_clean ($dir);
        $output .= qq|<a href="$sectionhtml#$precleandir">$dir</a>:|;
    }
    chop ($output);
    return $output;
}

Misc fixes and enhancements

May 17, 2002

Bug Fix: Updated and new image bugs squashed.
Bug Fix: Category columns now divided more evenly.

Enhancement: Added the ability to filter the research page by keyword. I think the description I wrote it clear enough, as it's not a search of the contents, just of the link names. I'll have to watch the logs and see if people understand....

Remaining bugs:
Special characters in fields are breaking the RSS and Javascript output. Need to figure this out.
Bounced emails from the newsletter are not coming back. Either there is something I set up wrong, something is wrong at my host, or its a result of the script running as "nobody". There was some discussion of this at the gossamer threads forums, but nothing definitive that I could find. Any ideas?

I'm no longer concerned about sorting the categories in non-alphabetical order, so the next public release of the mods will be after I figure these last bugs out. Let me know if you can help.....

Other still dreamed-for enhancements:
Get the what's new to produce a chronological history of what was added, and
To fix the newsletter sort order. Even better, I'd love to ensure that links are only sent to the newsletter once, therefore making the sort order irrelevant.

Newsletter introductory text fix

April 6, 2002

The introductory text does not belong in nph-email.cgi, rather it goes in admin_html.pl in the sub html_mail_update. Right after the TEXTAREA tag, put the introductory text you want. If that doesn't work, I'll try the more elegant solution discussed in the link below.

Gossamer Threads Forum

Add.cgi fixed

April 6, 2002

Add.cgi now works for categories. Until the next release, you can fix this yourself by editing site_html_template.pl. In sub site_html_add_form add Category => $category, before %globals

Note the comma after $category. This command was in the original Gossamer Threads code, so why I took it out, I don't know. Also note that some of the other fields (ie format) are hardcoded into the text, the email message doesn't contain all of the entered data, and some of the referrer information is missing from the email. But for our purposes it works. If you develop a more elegant solution, let me know and we'll combine it with our code.

New and updated image bugs

February 7, 2002

Currently the UPDATED images are absolute links to my server right in the code, and the new ones are relative links. This has drawbacks for portability and while it works for my main output, the new image links are broken in the search and probably other cgi-generated pages. The next revision will probably create a global variable for the absolute path to the image directory. This variable will be placed in the page generation code and in the templates for the search feature.

Newsletter not handling wrong addresses?

February 7, 2002

For some reason I don't get the bounced emails sent to bogus email addresses. It also appears to ignore the introductory message hard coded into the code. (Does it always ignore the introductory text? Not sure.) Feb 12 update: The introductory code is printing to the screen, but not the email message, so I must have the code in the wrong spot.

Apostrophes break javascript module output

February 4, 2002

My javascript export of new links isn't working all of a sudden. Turns out I have a headine with an apostrophe in it, and the apostrophe is used by javascript to signify the end of a field.

Apparently I need to create a clean subroutine to strip out apostrophes before generating the javascript. (Alternatively, can I escape a character in javascript like you would in perl?) As always, if you can get to this before me, please send the code.

Sort categories through non-english mod?

January 5, 2002

It might be possible to sort the categories through the use of the non english mod, which apparently creates a second field for the printable version of the category name. I wasn't able to find the mod in a quick (1 minute) search, but this might be the best way to go, although since I am not creating directories for each category, this might be more trouble than its worth.... Thoughts?

[Feb 1: I found the non-english mod. I think integrating this into the tree mod might be difficult and I might be better of doing it from scratch with that as a model.]

Somewhere I broke add.cgi

December 22, 2001

Something in these mods breaks add.cgi from displaying the category field properly. I suspect I mis-modified something related to the autobuilding of cgi forms. Looking at the code, it appears I added fields manually to the html and it can be done automatically. At one point is was also mishandling incorrectly entered fields, but that has not been tested in a while. This is not a key feature for me, but I'd like to get it working so that other non-admins can help build the database. If you figure it out, let me know.

A broken add.cgi in action

RSS Module Bug

December 17, 2001

The RSS mod I made does not encode special characters out of fields. Not sure exactly how to do that. Any ideas? It needs to display and our link properly when it comes out the other end. It appears that most readers will accept files with fancy database URLs, though.... (Is that because funky characters are ok in 0.92 which the readers but not my version of IE support? My file is pretending to be a 0.91....