We tested the STM600 over a one-month period for antispam effectiveness, Web security and e-mail configuration and features and performance.

To test antispam effectiveness, we used the same methodology we've used in prior antispam tests, running a production e-mail stream through the STM600 (and several other antispam appliances) for a period of about a week. The STM600 was configured with typical enterprise settings, although we did not use the quarantine. All told, we pushed about 15,000 messages through the STM600. We then collected the verdicts on each message and compared them with the verdict provided by a human judge, which let us compute spam catch rates and false positive rates. Because our methodology lets us separate out reputation filtering from content filtering, we collected statistics on each separately and combined them for a final score.

To test performance, we used Spirent WebAvalanche 2700 appliances to generate HTTP traffic, secure-HTTP traffic and e-mail.

For HTTP and HTTPS traffic, we set up a profile using a typical Internet mix of traffic, ranging in size from 1KB objects to 1.5MB objects, and ran HTTP transactions through the STM600 at a rate designed to place a load of up to 1Gbps throughput. Because the STM600 has two pairs of gigabit Ethernet interfaces, we spread the Web traffic load out across all four interfaces. Our traffic loads were designed to imitate an enterprise mix of different Web pages with image, text and binary data.

We ran tests offering a simultaneous load of about 2,000 open connections, and measured total throughput at the network layer. (Measurements at the application layer are actually more accurate and representative of what end users will see, but since most performance is reported at the network layer, we report our numbers in the same way.) We repeated each test three times, running one set for HTTP traffic and a second set for HTTPS traffic. Configuration defaults were used, except that antimalware scanning was enabled.

For the e-mail performance test, we again used the Spirent WebAvalanche 2700 and offered the STM600 a sustained load of about 100 messages a second. We measured the flow of messages accepted by the STM600 once it reached a steady state. Again, configuration defaults were used once antimalware scanning was enabled. We ran this test six times. One test, repeated three times, included a typical enterprise mix of e-mail, including spam, viruses and non-spam in approximately correct proportions. The second tests, also repeated three times, included only non-spam e-mails. The STM600 performed better with the spam and virus e-mail in the mix. Although Netgear did not share the internals of their e-mail pipeline with us, it is likely that it does spam scanning before virus scanning, because spam scanning is less expensive in resources than virus scanning. Since most e-mail is spam, this would let them short-circuit virus scanning for messages that are known to be spam and thus achieve higher performance. We reported results for the mix of traffic since we believe those to be more representative of what end users will see with this system.

Return to test.