Greylisting… Revisited

  by Steve Freegard

Our BarricadeMX products have supported greylisting since they were first released. From the beginning, our greylisting used custom modifications to make it more acceptable to businesses and to address some of the original method’s shortcomings.

As we have started working on what will become the 3rd version of the BarricadeMX SMTP engine; I started by looking back at support issues over the past few years to see what we could do better and decide what should and should not be present in the new version.

As part of that review process I started looking again at greylisting; should we keep it or should it go?  I had found a number of tickets where we had advised customers to whitelist servers because they did not handle greylisting at all.  But was this reason enough to get rid of it completely?  Is it still effective?

For the purpose of this paper, it is assumed that the reader has an understanding of the Simple Mail Transfer Protocol (SMTP), the Domain Name System (DNS), and related terminology.

Background

Greylisting has been around since 2003 when Evan Harris wrote the original
whitepaper on it as a spam filtering mechanism.

The basic premise was that during the SMTP connection for each connecting IP address, sender, and recipient a tuple is created in a database and the recipients deferred (with a temporary failure) for a fixed period of time.  If the same tuple comes back after that period of time had passed, then the recipients are passed and the message accepted.  The idea being that if the sending system implements a retry-queue it would pass the greylisting after a period of time, while spam engines would not because they do not implement retry queues, but instead work in a fire-and-forget fashion.

The issues with this approach were that using the proposed tuple of IP, sender, and recipient caused mail to be delayed excessively for a period of time until the greylisting system had ‘learnt’ who are the regular correspondents.  Worse still, SMTP servers can have wildly varying timeouts meaning some will retry after a few seconds, others might take hours to retry meaning that genuine mail can be delayed for very long periods.

The other issue was with SMTP implementations that use a shared spool for the retry-queue; this means that while one host might send the initial message, another host on a different IP might send the retry; therefore the greylist tuple will not match this retry again causing severe delays.

Many implementations were created with various workarounds for these shortcomings.  

Many people have spoken out against greylisting and have speculated that like all anti-spam mechanisms it will become less effective over time, because the spammers will adapt to it.   So in 2011, eight years after the original whitepaper was published, has this happened?

Testing

Because traditional greylisting sends a temporary failure to each recipient, quantifying how many messages this actually equates to is impossible.   So for the purposes of this test I will be sending the temporary failure at the end of DATA so that I may count unique messages and track the retries of each message.  This comes with a big caveat – bitter experience with SMTP implementations tells me that greylisting a message at the end-of-data phase instead of each recipient can and probably will cause different results in the real world, because some SMTP implementations handle retries differently depending where the deferral is done and what deferral code was used.

As noted in the ‘background’ section above, using the IP address as part of the tuple causes problems with shared-spools.  Many greylisting implementations therefore only use the first 3 octets of the IPv4 address to treat a single /24 as one entry.  The problem with this approach is that some shared spools might use more than one /24 or there might be multiple unrelated hosts in that same /24; either way this could also cause some undesired results.  To overcome the issues with shared spools I will use a different approach:

The full IPv4 or IPv6 address will be used if:

• The connecting host has no PTR record, a.k.a. reverse DNS (rDNS).
• The rDNS record contains the first two or last two octets of the IP address.
• The rDNS record contains the ‘short’, decimal, or hex representation of the full IP address.
• Multiple rDNS records are returned.
• The rDNS record cannot be verified by forward confirmation (e.g.
  FCrDNS).
• The top-level-domain (TLD) used is not valid.

Otherwise the first label, host part, of the rDNS is stripped, but only until the domain boundary will be used.  For example:

For the remainder of this paper I will refer to this hybrid IP / rDNS key as the ‘hostid’.

This method presumes that shared spool hosts will always have valid rDNS and that the pool of hosts will also either share a common sub-domain or domain.  Experience of using this method in our BarricadeMX products has proven this to be the case.  

Here is an example of this from the results, this shows a single message being received and greylisted and each of the retries showing the time difference between the last retry and the IP address of the host sending the message  In this case the hostid is ‘.obsmtp.com’ (Outblaze):

As can be seen, multiple retries from a pool of hosts sharing a spool are easily handled by one greylist ‘hostid’ with a total greylist delay for this message of 859 seconds.  My only comment regarding this is that the retry interval used here is rather short with no back-off.

The greylist tuple I am going to use for the purposes of this paper and so that I can track unique messages and their retries will be:

The other deviation from the original paper is to prevent excessive delays for each hostid.  All that greylisting proves is that a host (or group of hosts) correctly implements a retry-queue.  Once that has been proven then that ‘hostid’ is exempted from further greylisting for 40 days since it was last seen.  This is done to reduce the impact and delays that greylisting can cause.  Although I have not done so in this test I also recommend that hosts listed on the public DNS whitelists such as the DNS Whitelist (DNSWL) be exempted from greylisting for the same reason.

To prevent spammers from exploiting this exemption by continuously cycling through their sender, recipient, and template lists in a effort to defeat greylisting and/or be exempted, a ‘hostid’ must retry either the first or the last message that has been seen from that given ‘hostid’ (this is to handle last-in-first-out and first-in-first-out queue strategies), until that time any other messages will be greylisted.

I also had to choose what timeout periods I would use:

850 seconds for the greylist deferral period. This was chosen because a Wikipedia article on greylisting shows the default retry times for various MTAs with the average 1st retry being around the 15 minute mark, 850 seconds prevents a host retrying every minute from passing before this.

I also believe that 15 minutes is a good margin for a DNS blacklist (DNSBL) to see the message from an IP (e.g. infected, malicious, hacked or other), list it, sync the list with any mirrors and allow for a negative cache TTL to expire.  The theory being that if host is greylisted and is not already listed on a DNSBL, then prior to accepting mail from them, they could be subsequently black listed.  This theory maybe the subject of a future article.

25 hours before greylist deferral records are removed. This was chosen in case someone had set their queue retry timer to 24 hours, unlikely, but you never know.  I just wanted to pick an arbitrarily big number. I can revisit this choice based on the results in my conclusions.

Based on the 25 hours before any greylist records are removed, I decided that the test would run for a total of 50 hours.  The first 25 hours would allow any messages through as normal subject to the rules of greylisting, after which I would only allow through messages for hosts that had outstanding greylist records to give them the chance to pass the test, no other messages would be allowed or tracked after this time to allow the test to finish gracefully.

I should also mention the mail stream that I am running this test on: it’s an old 3-letter domain that was first registered in 1989, but has been out of use for the past 10 years or so and has no real users present on it.  I spent a lot of time generating addresses that were harvested and sold on a few years ago, so it gets a quite a lot of botnet spew, mainsleaze and the occasional bit of misdirected mail.

Results

Lost is the count of the number of messages that were not retried after the hostid that sent them had passed the greylisting test.

These results were quite a surprise to me.  Prior to starting this test I expected the numbers to be considerably lower.  With this in mind I decided to analyse the results further to verify the outcome.

I started by analysing the number of retries seen for each message for all messages that had been seen during the test.  The graph speaks for itself:

It shows that nearly all messages that failed greylisting made only a single delivery attempt.

For comparison – here is the same graph, but showing only messages that passed greylisting:

And here is a graph that shows the maximum delay caused by greylisting:

This shows the vast majority of messages were initially delayed for less than an hour.

Extremes

The graphs above show some big numbers at the extremes.  I decided to investigate these.

• 61 failed retry attempts.  All of the retries counts over 17 were caused by a single group of hosts with a hostid of ‘newsletter.m6.fr’.  It would appear to be a pool of hosts with a shared-spool and a retry interval of 10 seconds.
• 17 retry attempts for a message that passed greylisting. This was a pool of hosts with a hostid of ‘.xraybot.com’ running a shared-pool that appear to be running the Lyris ListManager 10 software (verified by visiting
  www.xraybot.com and the SMTP banners).  

  In this case there are 16 hosts in the pool and each host retried 1 second after the previous host, then once all of the hosts in the pool had attempted to send the message – the first host in the pool retried an hour later therefore passing the greylisting.

• A message that was delayed for almost 24 hours. This was from a single host with a hostid of ‘.built2go.com’ and an envelope from ‘root@unknown.scnet.net’, but the From header shows ‘updates@boxedartupdates.com’.  It scored 1 in the SpamAssassin mass-checks and appears to be a non-spam newsletter.  According to the headers, the message was delivered by Sendmail, so it was simply configured with a very long retry.
• Not shown on the graph, but in the greylist database I found a single case where the retried message was over 25 hours old and the greylist record had been deleted.  The message was a lottery scam message relayed through host
  dormant.esatclear.ie via authenticated SMTP from a host in the Czech Republic and was retried after 28.9 hours.  According to the received header the host was running Exim 4.14.  The message would have passed greylisting had the record expiry time have been higher.  I the only explanation I have for the very long retry timer is that the host is under heavy load and is not running its queues due to that load.

SpamAssassin Mass Checks

I ran a SpamAssassin mass-check for each message class that I had collected from the test.  This produced some interesting results, but it should be noted that the mass-checks were run after the test was complete.  That means that some of the messages checked could have been over 50 hours old and would most likely have scored lower at the time the message was received.

I started with all the messages that had failed to greylist properly:

This shows 0.6% of the messages were underneath the SpamAssassin threshold of 5.  I manually inspected these messages and found only two that I would consider a false-positive, the rest were all false-negatives.

Both of the false-positives were a newsletter from the New York Post (Subject: Daily Newsletter).  Upon inspection both of these messages were only attempted to be delivered once.  The Received header shows that Postfix was the MTA used to deliver the message, so it must have been configured to do this is not the default behaviour.  The messages were sent from IP 209.73.248.15, which is listed on the DNSWL.

The mass-check results also show a high overlap with DNSBL / URIBL services:

Here is the score distribution of messages that were not retried after the host sending the messages had passed the greylisting test with another message.  As there were only 34 of these I manually looked at all of them and found them all to be spam.

I would speculate that some of these messages were not retried as the hosts were found to be compromised and the messages were subsequently deleted from the queues.  On previous trial runs of this test I found evidence that Google did this when it shut down compromised accounts.

Again I found a significant overlap with DNSBL / URIBL services in the mass-check results:

And finally here is the score distribution of the messages that passed greylisting:

As the mail stream I’m testing this on predominantly receives spam, it was not much of a surprise that a high proportion of these messages were considered to be spam.  Here is the overlap with the DNSBL / URIBL services from the mass-check results:

Overheads

The database that I used for this test recorded a single greylisting record for each greylist tuple which corresponds to a unique message, and a separate table tracked the retries for that tuple.  This allowed me to track the bandwidth overhead associated with greylisting at the end-of-data in this way.

If all messages greylisted during this test were allowed through without being greylisted, the total size of the message data accepted would have been 151 MB excluding any protocol overheads.  With greylisting this rose to 210 MB which is a 39% increase.  However greylisting prevented 130 MB of traffic from being content scanned, an intensive activity, which is an 89% decrease in volume.

Queue strategies

In the ‘Testing’ section above, I mentioned the requirement that a hostid should not be able to pass greylisting until it had retried the first or last message (e.g. LIFO or FIFO) that it had sent to prevent spammers from passing the greylisting via a ‘brute force’ retry method.  This would also allow an optimisation to reduce the amount of bandwidth used by moving the greylisting deferral back to the RCPT TO: or DATA stage if the sender and recipients do not match either the first or last greylist tuple for that hostid.

However, while doing initial testing and removing any bugs from my greylist plug-in, I observed that several messages from free-mail hosts were ‘lost’ and never retried after the hostid had passed the test.  Upon investigation, these messages were all spam and I concluded that they were never retired because the abuse was detected and the accounts that sent them were either disabled or deleted and any messages in the queues for these accounts were de-queued.

If one of these messages deleted from the queue had been either the first or the last message seen by the greylisting plug-in, then all mail from that hostid would have been delayed considerably as the hostid would not be able to pass the test for over 25 hours (in the case of a FIFO queue strategy).  I also realised that I commonly ran Sendmail queues on overloaded systems with ‘sendmail -qp -OMinQueueAge=15m -OQueueSortOrder=random’ which would also fall foul of this requirement.

The data collected for this paper showed that only 6 messages were rejected because of this requirement that would otherwise have been accepted.  All of the messages were considered to be spam by me and by SpamAssassin.

Based upon these observations I would not recommend making the queue strategy a requirement to pass greylisting at this time.

Conclusion

The numbers I think speak for themselves, even after eight years greylisting is still effective.  With modifications to the greylist tuple used, such as the hostid, exempting hosts that finally pass greylisting, and skipping greylisting for hosts on DNS whitelists it can be effective without adversely delaying genuine mail.

It can also significantly reduce the amount of external network look-ups required and reduce the CPU cost of content filtering until a host has passed greylisting allowing greater scalability at the expense of extra bandwidth.

Critics of greylisting as an anti-spam technique would say that it can cause valid mail to be lost, I would counter this argument and say that any e-mail server or web application that cannot correctly handle temporary failures as required by the RFC is going to be terminally unreliable anyway.

While it might not be for everyone and it will still require some hosts to be manually exempted. Based on these results greylisting will definitely be in the next version of BarricadeMX and our customers can choose for themselves if they want to enable it or not.

Thanks

The original idea for greylisting using part of the PTR record was made by Anthony Howe.  My thanks to Stephen Swaney, Randolph Langley and Anthony Howe for editing and contributing changes to this paper.

Copyright 2011 by Fort Systems Ltd. All rights reserved.
May not be reprinted without permission.

From: Steve Linford at Spamhaus
Date: Sat, 18 Dec 2010 12:39:18 +0000

For speaking out about the crime gangs located at the wikileaks.info mirror IP, Spamhaus is now under ddos by AnonOps.

As our site can’t be reached now, we can not continue to warn Wikileaks users not to load things from the Heihachi IP. If you know journalists who would get this message out, please forward this message (entire) to them.

AnonOps did not like our article update, here’s what we said and what brought the ddos on us:

—-

In a statement released today on wikileaks.info entitled “Spamhaus’ False Allegations Against wikileaks.info”, the person running the wikileaks.info site (which is not connected with Julian Assange or the real Wikileaks organization) called Spamhaus’s information on his infamous cybercrime host “false” and “none of our business” and called on people to contact Spamhaus and “voice your opinion”. Consequently Spamhaus has now received a number of emails some asking if we “want to be next”, some telling us to stop blacklisting Wikileaks (obviously they don’t understand that we never did) and others claiming we are “a pawn of US Government Agencies”.

None of the people who contacted us realised that the “Wikileaks press release” published on wikileaks.info was not written by Wikileaks and not issued by Wikileaks – but by the person running the wikileaks.info site only – the very site we are warning about. The site data, disks, connections and visitor traffic, are all under the control of the Heihachi cybercrime gang. There are more than 40 criminal-run sites operating on the same IP address as wikileaks.info, including carder-elite.biz, h4ck3rz.biz, elite-crew.net, and bank phishes paypal-securitycenter.com and postbank-kontodirekt.com.

Because they are using a Wikileaks logo, many people thought that the “press release” was issued “by Wikileaks”. In fact there has been no press release about this by Wikileaks and none of the official Wikileaks mirrors sites even recognise the wikileaks.info mirror. We wonder how long it will be before Wikileaks supporters wake up and start to question why wikileaks.info is not on the list of real Wikileaks mirrors at wikileaks.ch.

Currently wikileaks.info is serving highly sensitive leaked documents to the world, from a server fully controlled by Russian malware cybercriminals, to an audience that faithfully believes anything with a ‘Wikileaks’ logo on it.

Spamhaus continues to warn Wikileaks readers to make sure they are viewing and downloading documents only from an official Wikileaks mirror site. We’re not saying “don’t go to Wikileaks” we’re saying “Use the wikileaks.ch server instead”.

—-

Steve Linford
The Spamhaus Project
http://www.spamhaus.org

Update: The full Spamhaus warning concerning wikileaks.info is here. A similar warning made by Trend Micro is here.

SnertSoft is happy to announce Private Wimp, a simple light weight mailing list manager, that is free to download. Private Wimp has been used to manage several mailing lists for past few months now.

It avoids the bloat of Major Domo, Mailman, and Ecartis . Completely written in C and so avoids the overhead of Perl or Python. Installation and configuration is short and simple; all list management can be done remotely by email; always confirms subscribe/unsubscribe requests and admin. commands; handles bounce messages (discard, forward, or removal of unknown users); provides support for four list types (announcement, trusted, closed moderated, open moderated); keeps track of users that unsubscribe; and provides a simple archive structure (similar to Ecartis).

The online documentation can be found here:

http://www.snertsoft.com/sendmail/wimp/

I even wrote Javascript classes that implemented them. I thought it would be kind of neat in an odd ball off the wall sense, like expressing the speed of light in furlong per fortnight, to display them here on the blog (see sidebar right). I’ve even had an idea for a new RFC

However, I have two issues with this:

First, to achieve this data gathering, a web site is required to load on each web page of interest a Javascript file called urchin.js from Google or the more advanced ga.js file. Essentially a web site is telling your browser to execute some remote 3rd party script on your system. This is a BAD idea in terms of security, since it might be possible to hijack that script in transit and replace it with attack / hack code. Also the script is not loaded securely via HTTPS, so no certificate authentication or validation of any kind is done; just blind trust that google-analytics.com has not been hijack by DNS cache poisioning or that some intermediate web proxy hasn’t been compromised.

Second, I am interested in protecting my privacy online as much as possible these days. I already have a pretty big online foot print dating as far back as 1986; regardless I see it as my right to restrict data collected about me. So whenever a web site asks for HTTP cookies, Flash Cookies (How to Manage Flash Settings), tries to load advertising, or track my movements through scripts and/or cookies, I’ll go out of my way to block that from happening.

So when a web site loads urchin.js or ga.js, it is going to communicate information about visitors back to Google. I find this an invasion of my online privacy. What I do online is my business, not Google’s. Google already has enough data about what search terms I look for (this can be controlled through Google, though who knows if it is honoured or not). Frankly I don’t think Google or any other 3rd party advertiser needs to know where and what the frack I’m doing.

Simple solution: use a URL blocker, like Bork Bork Bork! or Adblock Plus, to block urchin.js, ga.js, and/or anything from google-analytics.com from being accessed. If you don’t want to use a URL block, this can also be achieved by adding to the Unix or Mac OS X /etc/hosts file (Windows has an equivalent C:\WINDOWS\system32\drivers\etc\hosts) and add an entry like:

127.0.0.1  www.google-analytics.com

Most webs sites where google-analytics.com has been blocked are designed well enough to continue functioning. However, there are a small handful of web sites the refuse to do anything when the tracking code is not loaded. Typical bad design on the web sites part. In the end I see Google Anal-Tics as evil and chose not to do business with web sites that expect me to put up with that shit.

My Muzeella Fureffux & Thoonderburd ixtenseeun Bork Bork Bork! hes beee updeted fur Muzeella Fureffux 3.5. Bork Bork Bork! is a Svedeesh Cheff trefesty feelter und URL blucker. Feeoo veb peges oor (joonk) meeel es spukee by zee Svedeesh Cheff leeke-a thees.

It is available directly from:

• my snert.com page for Bork Bork Bork! 1.8
• the official Mozilla Add-On page for Bork Bork Bork! 1.8
  (if they ever get around to approving it)

Sites like PHP.net and Flickr have already added support for many of the ideas discussed. But what is really required is that there be more adoption by blogs, social network sites like twitter and identi.ca, and applications like twhirl and tweetdeck.

To that end there is already a WordPress plugin called Shorter Links, which having installed it here works very nicely and I assume the author will continue to track the developments in this space. There is also a tool to test self-published shortened URLs.

One can also follow the twitter discussion thread about #revcanonical.

• Malwarebytes Anti-Malware – the best malware detection tool I’ve used to date that works! Will catch things that an up to date AV, adware, and spyware scanners fail to find. If you suspect you have a problem, get this tool, update it, and do a quick scan. Odds are this tool will save you from a tedious system wipe and rebuild.
• Avast! Home Edition – free for private and personal use. This has been the one I’ve been using for a couple of years now. See AV Comparitives.
• NOD32 – commercial AV with free trial, not yet tried this, but I’m told by a fellow sys.admin. friend that swears it is the fastest, least resource consuming. See AV Comparitives.
• Vipre – another colleague suggested last night this AV scanner, but it appears to be completely unknown underdog. But I trust the source of the suggestion.
• ClamAV for Windows – free open source anti-virus scanner; however it lacks an on-access scanner, which is essential for being alerted to problems quickly. Have tried to find add-on on-access scanners for ClamWin, but not yet found a suitable one. ClamWin is great for whole disk scanning though, but with modern disk being so big, how often do you bother to scan a whole disk or individual files.
• System Internals Tools – bought out by Microsoft, they have a superior Process Explorer, and many many other neat power user / admin. tools.
• Tweak UI – part of the Microsoft Power Toys suite and essential for customising Windows behaviour, like turning off the annoying “Ballon Tips” or disabling CD/DVD autorun to prevent installation of the evil PC Friendly (causes nothing but grief) or other potential nasties the studios try to slip onto a machine, like DRM root kits.
• Registry Guide – formerly regedit.com and winguides.com, they used to provide a Windows helpfile download showing many many handy registry keys, but now it’s only available online (grrr). Documents much of what you can change using TweakUI or regedit.exe. Handy information for locking down a Windows computer. Here’s an out of date copy of the last free Registry Guide downloadable.
• SiSoftware Sandra Lite – everything you wanted to know about your computer hardware.
• Memory Testing – A comprehensive free memory testing tool.
• SpeedTest – a handy bandwidth testing web site. BTW it helps if you know where your network provider’s “peering” is done in order to compute favourable results.
• Ranish Partition Manager – a free tool for resizing and managing primary & extendied disk partitions. Handy if you want to setup dual boot systems.
• Admin. Password Reset Tool – have you ever forgotten the admin. password for your Windows system or have you ever had to service someone’s machine to remove virii and needed admin. access.
• Treesize Free – handy tool for seeing what the size of directories are and where you might be wasting disk space. Also handy for estimating CD/DVD backup sizes. I have a copy of the older TreeSize Pro 2.4 which is just brill.

When it comes to AV tools, I’ve given up on Symantec and McAfee. I think they’re past their heyday. Symantec Norton Anti-Virus is a resource pig that can slow a Windows machine down at least (I estimate) 20%, certainly noticeable; the user interface is slow; and frankly it misses catching virus, trojans, spyware, etc. In my humble opinion its rubbish. As for McAfee, I stopped using it some where around Windows 98, when it just stopped being as affective in identifying malware. At the time I switched to Norton AV and was happy for a long time until Windows XP and performance problems started appearing. I’ve not revisited McAfee since, but frankly if I’m going to pay for an AV, I’m going to trying something different, like NOD32.

Forget about installing adware or spyware detection tools; remove them if you have. Frankly I do not trust these tools to not be the actual source of adware and spyware themselves. This should be the job of a good and well known anti-virus scanner. The only tool I’ve come to trust that I’ve seen catch stuff that an AV scanner have missed has been Malwarebytes. I recommend running this even if you have an AV scanner.

The above are just some of the handy tools I’ve kept booked marked for emergencies or use on a daily basis. I have others I could probably mention, but for Windows sys.admin. and field support the above is a good place to start and should keep you calm enough to get the job done. You’ll still curse Windows as rubbish, but at least it you might be able to fix it enough to tolerate it longer.

Update 2009-10-07:

• Avira AntiVir Personal – free for personal use. I’ve been using this for the past 10 months as I’ve found it to be less resource intensive (aka faster) than Avast! and just as good. It lacks many of the extra features of Avast!, such as SMTP, POP, IMAP, P2P, IM, and web scanning, but then for power users who are aware of the pitfalls, use secure channels, and use tools already adapted to their situation, like Firefox web browser, then Avira’s light weight nature compared to Avast! will be better. Still for the average joe unfamiliar with internet security, Avast! Home Edition will probably be a more comprehensive solution.

Along with the usual plethora of speed, accuracy, and bug fixes that are part of any major release, are several new features and enhancements:

Enhanced Message-ID for Email Watermark (EMEW) Version 2

Improved outbound message “water-marking” reduces the threat of Denial of Service due to “bounce message” floods. With EMEW it is now possible to selectively apply different secrets by individual sender, sender domain, or sender account for outbound tagging and validation of of inbound non-delivery reports or content white listing of replies. This allows an ISP to apply EMEW only for those domains known to use the ISP outbound mail servers exclusively and exclude those domains that might use a mixed mail server model.

Attachment Reject Policies

Using simple file name patterns, deny attachments based on attachment
name, content-type, and/or file names found in .zip and .rar compressed
archives.

Time limited recipient addresses

Easily generate safe and disposable time limited email addresses as part of user’s regular mail address. Intended for use by users who want to supply short lived addresses to questionable web sites registration forms or mailing lists.

Digest DNS Blacklist Support

Originally intended for use with the Malware Hash Registry, it can be used with other similar blacklists. Support for other distributed hashes, such as Vipul’s Razor, Pyzor, and DCC is being considered.

Sophos AV Support

Sophos AV has been added to the already supported AV engines: Avast, ClamAV, F-Prot.

More RFC Supprot

RFC 1652 8BITMIME simple pass-through support now advertised with EHLO capabilities list.

RFC 1870 SMTP SIZE parameter extension supported and can be used in conjunction with the existing access-map size limitation tags length-connect:, length-from:, and length-to: for rejections based on SIZE at RCPT TO: command instead of end of message.

DNS, URI, and NS BL Additions

Now possible to check IP addresses and URI found within selected headers against blacklists. Also experimental options to check URI name servers against specialised NS blacklists now available.

Technical questions concerning the software or documentation, please contact me via SnertSoft directly. Otherwise to arrange for demos, discuss pricing, speciality needs, or other POSIX based platforms please contact my partners FSL.

I have a theory if you anticipate the worst, things typically work first go, but you waste lots of time and energy being paranoid; of course that one time you’re not paranoid or too complacent about installing an upgrade happens to be that one time when the shit hits the fan. The same is true with computer hardware; when you install a new interface card or memory and carefully close up the box, before testing the new hardware, you always end up having to reopen the box because: the card / memory wasn’t seated properly; you unseated a cable by accident that just happened to get caught on your sleeve cuff button; forget to reconnect a cable you disconnected so you get your hands inside, etc. Sometimes I think Murphy was the court jester to God.

I’m Canadian and live in France. I speak and read both English and French, but English is my first and preferred language, especially for all things technical. I’ve configured my web browser Firefox (Opera and IE have this facility too) as to which language variants I want. The HTTP/1.1 web protocol as described by RFC 2616 supports the Accept-Language header that the web client software specifies in HTTP requests as to which language the user wants to receive in order of preference.

So !WHY! is it that web sites like Google, YouTube, and many others select a web page language based on the user’s geographical location (determined by country assignments of IP addresses) rather than my personal preferences!?! Especially when there is a protocol mechanism to facilitate language choice! Why should I then have to change the web site preferences and store a language cookie (RFC 2965) to remember that choice, when my web browser keeps telling the web site my preferences as part of each request I make where ever I go!?

Why do web sites insist on pissing users off by making broad assumptions about as simple a thing as preferred language? “Oh! You live in France, you must speak French by choice. We’ll give you the French version of the site.” Bzzzzt! WRONG! Game over! Thanks for playing! Bloody wankers! (I can make similar comments about language selection when installing software, my region is set to France, but I have a UK QWERTY keyboard! What does tell you about me? Grrr.)

My second peeve concerns Contact Us links on web sites, either the lack there of, that they are often buried deep deep in the web site in some obscure corner of a page, the poor choice of options such as no means to make general comments, suggestions, or ideas, or that the page is inaccessible or won’t display at all. I wanted to comment on my language selection peeve to YouTube, but there was no link for comments, just how to complain about copyright, abuse, security, get API information, and the like. Trying some of the alternative choices, like Help Centre, would not even display at all in the browser – as though the web page request was stuck in some sort of redirection loop.

One thing YouTube/Google have done is publish their postal address and phone numbers, so I’ll probably print a hard copy of this rant and mail it to them. If I had a fax (OK, I could use the computer’s fax service I suppose), I might do it that way, but I’d probably find their fax machine connected to an automated telephone system menu that I’d have to navigate first before I could get a carrier tone. Hmm. Maybe if I press zero for an operator and blast the modem tones in their ear. That might give me some small measure of pleasure and assuage my need use a clue bat on someone.

There is an old Internet saying my server, my rules or more precisely with respect to the topic at hand my blog, my rules. If I own, operate, and pay for the hosting of a web site or blog, then any content generated on that web site belongs me, thus comments left on my blog would belong to me, with proper attribution to the commenter of course (otherwise why bother having comments at all). If someone wants to save and/or protect their comments, then they should use their own blog and link back to the source article or comment to form the thread of discussion for readers.

My blog, my rules is a question of simplicity and easy of management of content. To do otherwise would be chaos.

What about issues of liable? The Internet has already seen cases where blog owners linking back to defamatory articles have been held liable, even though they did not write the original article. As a person running a blog, I need to be able to manage the content, especially when someone else contributes commentary, in order to protect myself legally. I do not think blog owners will have the same protection as an ISP or social network site (ie. America’s “safe habour” provisions). What happens when the blog owner and the commenter live in different countries, each with different laws with respect to freedom of speech and copyright?

Consider too comment spam. If commenters had rights to their utterances on blogs and you delete or edit a comment that is spam, would you be liable in some manner? A blogger has more risk in terms of hosting costs, what they publish, and general reputation, such that they must have ownership and control of what is said on their sites, especially when it is an opposing opinion. Commenters are more like hecklers in a comedy club audience.

Probably the best solution is to disallow blog comments altogether. Force people to remain silent or use they’re own blogs or web sites to voice their views in response to articles. At least then each blogger takes equal responsibility for what is said in public.

For example Reason 13 “lack of tools” I disagree with. My design tool of choice is a good text editor like TextPad or nvi. I never use an IDE as I find they get in the way of “how I think” about a problem and often hide the language preventing a programmer from learning a language properly. Outside of an editor, compiler, interpreter, and debugger all the additional tools that serve as programming aids can often be a crutch that impede a developer from actually thinking about design and logic; sure they can help, but I don’t see this as the reason behind a language not being adopted.

Reason 10 “hanging about” I disagree with. Java evolved out of C, C++, and Smalltalk. Algol inspired Pascal which begot Modula2 & 3 which in turn inspired Ada, which was sponsored by US military needs. Lisp inspired Scheme and Lisp is still the favourite of the Emacs crowd. Python draws some of its roots from ABC and Fortran in terms of using indentation / column position for code blocks. Forth, a stack based language, was probably the inspiration for Postscript, which are both popular in their respective domains (embedded systems, printing). The point here is that the older languages had merit in their time, but go on to inspire a newer variant, and some languages have a certain niche domain or problem space that they were design for, rather than being general purpose.

Reason 9 “lack of sponsor” I disagree with. Perl became popular because it provided superior regular expression handling as syntactical objects, instead of a set of functions or methods. In addition Perl brought together all the elements of AWK, sed, and grep (regular expression based tools) and the shell into one kitchen sink type language at a time when shell script programming was painful due to the multitude of shells (Bournce, Korn, Csh, …) and their assorted quirks. Don’t recall there being any particular big name sponsor behind it. I’ve read some where that Larry Wall would not have written Perl had Ruby existed at that time to solve his needs. This would imply that the adoption of a language by programmers is also a matter of timing in addressing a need.

Also I see the creation of huge collections of libraries that are distributed as part of the core support more of a discouragement to learning a language like Java, Perl, or Python. I dislike “everything and the kitchen sink” approaches. I learnt Java several years ago before the sudden influx of APIs where added. It was appealing then because one could see and understand the whole of it. Now I seldom program in Java, because I just see it as bloated mess; the language is ok, but all the additional libraries that are bundled with it now make me want to run screaming into the hills.

Reason 8 “no killer application” I strongly disagree with. A killer application is a new and novel idea or concept, which is separate from the implementation language. The choice of language used to develop a killer application might simplify parts of the implementation, but do not inspire the actual ideas behind the application. What a killer application does for a programming language is give it exposure. However, this can lead to programmers “worshipping the messenger more than the message”, which is why some languages like Perl, Python, Tcl, or Ruby on Rails will gain sudden followings. What people do after with their language(s) of choice will be more interesting to history.