Why Signature Detection Fails.

The problem with almost any intrusion detection system are unknown attacks or unknown vectors. Likewise anti-virus software has difficulty protecting against all of it. There are many tricks to fool intrusion detection or AV software. One of them is generating false alarms or increase the noise on the line. Another technique is to make a reverse (shell) connection to a webserver or client. This can be of use to bypass certain firewall rules or intrusion detection mechanisms. Intrusion detection or firewall rules can detect obfuscated shellcode, but this isn't perfect. There are plenty of ways to obfuscate shellcode or other attack vectors. In fact there are so many ways, that it is impossible to detect them all. Hexcode can be translated to unicode giving it a trusted look and feel, ip addresses can be presented in a myriad of ways by uses octal dotted, decimal, (d)word, or even hexadecimal dotted conversion. Just like I talked about last year when I described how to bypass the Firefox anti phishing filter with these techniques; see here.Blacklists don't work, it is about time we cut to the chase and accept that. It is as simple as that. But I hear you, how can you whitelist a vector that was obfuscated to be allowed? like converting hexcode to unicode, which has whitelisted representational characters in them? clever question I must say! the solution isn't whitelisting alone, security is a process that requires also to anticipate upon the expected character set. This can mean that we also whitelist only UTF-8 for example, before we filter our secondary whitelist rules.The reasons for this speaks for itself. As seen in the last few months, it is nearly impossible to blacklist everything we can use in Javascript. Sure, we could continue in this effort, but the only thing that will change is the size of your rules and regular expressions, making your code slow and still vulnerable to unknown attacks. To mitigate unknown attacks, among other things you have to consider whitelisting in combination with apriori character set detection, otherwise you'll be playing an arms race that cannot be won.To close the article, here is a good example done by Gareth Heyes: