ISPs and large enterprises are being offered a novel way to stop spam that goes beyond the mere filtering of e-mail messages - detect and block the botnet zombies that generate much of the problem in the first place.

Mail security vendor Engate claims that the new version of its MailSentinel gateway, version 3.6, expands its anti-bot rules database to detect activity in real-time at the protocol layer using a mixture of proprietary traffic analysis, source verification, and anti-forgery techniques.

The new features are mostly in the areas of source verification and anti-forgery, important because botnet designers now go to some lengths to evade detection.

The system profiles the whole network, working out which IP addresses are legitimate servers and which are ordinary PCs and should not be sending large volumes of mail. If a client changes its behavior, as would be the case after an infection, MailSentinel will detect the change and block the client's traffic.

According to Engate, because the system works at protocol level, it can also be extended to cope with instant messaging, VoIP and mobile applications as well as SMTP e-mail.

"We use a variety of specialized tools and proprietary techniques to identify the function of every IP address in the network and we create efficient rules to block connections coming from illicit MTA IP addresses (bot clients) and allow connections from legitimate MTA IP addresses to pass," explained Engate's Tony dellaBusa.

"Once a new IP address is compromised, we'll already have their profile and we're able to immediately detect this compromised source as it emerges and pre-emptively stop it from transmitting spam and malicious payloads at the network level," he said.

Importantly, however, while the system blocks botnet client traffic it does not actually do anything about the infected client itself, which will continue to generate spam. But it can tell an admin or ISP where the problem lies.

Anti-botnetting has grown in popularity in ISP gateways for obvious reasons, but it's still relatively hard to separate legitimate activity from botnet traffic. On enterprise products, anti-bot filtering is still at the leading edge of mail and traffic security - most mail gateways look only at content and don't perform extensive analysis of traffic patterns. MailSentinel is a sign that this might be starting to change.

More information on MailSentinel , including pricing, can be found on the company's Web site.