Michael Rash, Security Researcher

2010 Blog Archive    [Summary View]

How to avoid ClamAV matches on bundled Snort rules

ClamAV matches on Snort rules Recently a psad user notified me via email that the psad-2.1.7.tar.gz tarball was flagged by ClamAV as being infected with Exploit.HTML.MHTRedir-8. After checking a few things, it turns out that ClamAV is triggering on a Snort rule in the Emerging Threats rule set which is bundled in both psad and fwsnort. The following analysis shows exactly what ClamAV is detecting and why, and also provides some guidance for how to avoid this for any software projects that distribute Snort rules. Similar logic would apply to other software engineering efforts - including commercial intrusion detection systems - that are (by their nature) looking for malicious artifacts on the filesystem or within network traffic.

First, let's download psad-2.1.7.tar.gz and check the gpg signature (just to make sure we're talking about exactly the same file): $ wget --quiet
$ wget --quiet
$ gpg --verify psad-2.1.7.tar.gz.asc
gpg: Signature made Wed 14 Jul 2010 06:01:42 PM EDT using DSA key ID 0D3E7410
gpg: Good signature from "Michael Rash (Signing key for projects) <>"
Ok, now here is what clamscan says about the psad-2.1.7.tar.gz tarball: $ clamscan psad-2.1.7.tar.gz
psad-2.1.7.tar.gz: Exploit.HTML.MHTRedir-8 FOUND

----------- SCAN SUMMARY -----------
Known viruses: 816934
Engine version: 0.96.1
Scanned directories: 0
Scanned files: 1
Infected files: 1
Data scanned: 6.42 MB
Data read: 1.16 MB (ratio 5.55:1)
Time: 7.169 sec (0 m 7 s)
Let's see which file within the psad-2.1.7 tarball matches the Exploit.HTML.MHTRedir-8 signature: $ tar xfz psad-2.1.7.tar.gz
$ clamscan -r -i psad-2.1.7
psad-2.1.7/deps/snort_rules/emerging-all.rules: Exploit.HTML.MHTRedir-8 FOUND

----------- SCAN SUMMARY -----------
Known viruses: 816934
Engine version: 0.96.1
Scanned directories: 41
Scanned files: 405
Infected files: 1
Data scanned: 12.55 MB
Data read: 6.41 MB (ratio 1.96:1)
Time: 8.446 sec (0 m 8 s)
Intuitively, this makes sense. That is, given that ClamAV is out to identify nasty things within files, and given that Snort rules are designed to identify nasty things as they communicate over the network, it stands to reason that there might be some overlap. This overlap is not an indication of something wrong in either the Snort rules or in ClamAV. Now, let's find out specifically which Snort rule within the emerging-all.rules file is triggering the ClamAV match. We first take a look at the Exploit.HTML.MHTRedir-8 signature: $ cp /var/lib/clamav/main.cvd .
$ sigtool --unpack main.cvd
$ grep Exploit.HTML.MHTRedir-8 main.ndb
The last line above is the entire ClamAV signature, and the pattern 6d68746d6c3a66696c653a2f2f is the key. The ":3:" part identifies the signature as type "normalized HTML", so ClamAV matches the pattern 6d68746d6c3a66696c653a2f2f against the "normalized HTML" representation of each processed file. We can decode the pattern as follows: echo 6d68746d6c3a66696c653a2f2f | xxd -r -p
So, within the emerging-all.rules file, we are interested in any Snort rule that contains the string mhtml:file://. There is also the "{1-20}2168" criteria which says to match the hex bytes 2168 anywhere from 1 to 20 bytes after the first pattern match. $ grep mhtml psad-2.1.7/deps/snort_rules/emerging-all.rules
alert tcp $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS (msg:"ET MALWARE Bundleware Spyware CHM Download"; flow: to_server,established; content:"Referer\: ms-its\:mhtml\:file\://C\:counter.mht!http\://"; nocase; content:"/counter/HELP3.CHM\:\:/help.htm"; nocase; classtype: trojan-activity; sid: 2001452; rev:4;)
Sure enough, sid:2001452 "ET MALWARE Bundleware Spyware CHM Download" has the keyword content:"Referer\: ms-its\:mhtml\:file\://C\:counter.mht!http\://". Even though there are escaping backslashes, the normalized HTML processing in ClamAV takes this into account and matches the pattern anyway from the ClamAV signature.

So, how can we keep the original Snort rule, but change it so that ClamAV not longer flags it?

Fortunately, ClamAV does not interpret the Snort rules convention of specifying non-printable bytes between "|" characters within content fields, so we simply need to change one of characters to hex notation. Snort will still offer the same detection if network traffic matches the rule, and ClamAV won't flag it. So, let's just change the "m" in "mhtml\:file\://" to its hex equivalent, like so: "|6d|html\:file\://". Once we make this change and save the psad-2.1.7/deps/snort_rules/emerging-all.rules file, we rerun clamscan: $ clamscan -r -i psad-2.1.7

----------- SCAN SUMMARY -----------
Known viruses: 816934
Engine version: 0.96.1
Scanned directories: 41
Scanned files: 405
Infected files: 0
Data scanned: 17.23 MB
Data read: 6.41 MB (ratio 2.69:1)
Time: 10.024 sec (0 m 10 s)
That's better. Over the next few days I'll re-release all affected versions of psad and fwsnort with the above change to ensure that there aren't any additional ClamAV matches.

In conclusion, if you are involved in any software engineering effort that distributes or makes use of Snort rules, it is probably a good idea to run distribution packages through ClamAV and see if there are any matches. If so, it may be possible to take advantage of Snort rule syntax options to still achieve the same signature coverage while not having ClamAV flag anything.

A new SSH password guessing botnet: dd_ssh

dd_ssh botnet Every few months it seems that there is a new variation of a password guessing botnet. This time around, as reported by the SANS ISC and also by, a botnet called "dd_ssh" has been built up around a vulnerability in phpMyAdmin. (Here is the vulnerability announcement from the Debian security list.) Once a vulnerable webserver has been compromised, the botnet starts scanning random IP addresses for SSH servers and attempts to brute force passwords for those systems that accept an SSH connection. Although software like Fail2ban and DenyHosts can offer detection and mitigation by applying thresholds to the number of failed login attempts in the SSH log and communicate offending source IP's to a set of interested users, a fundamental problem is that the targeted SSH daemons are reachable in the first place from arbitrary source IP's. A more powerful solution is to hide your SSH daemon behind a default-drop firewall policy and only grant access through Single Packet Authorization. Any botnet that is scanning for SSH servers will not be able to guess any passwords or even exploit a zero-day vulnerability in an SSH daemon that is protected in this way.

Further, SSH brute forcing botnets have shown some innovative techniques where password guessing attempts are spread out over a long period of time and failed passwords are distributed across the botnet. The end result is that each botnet node knows the population of passwords that other nodes have already tried on each targeted SSH daemon, and so new passwords can be tried while still slipping beneath common detection thresholds. The same password never has to be tried twice botnet-wide. It is not clear yet whether the dd_ssh botnet uses a similar password distribution strategy, but either way I intend to stop such guessing games at the earliest stage possible and block all SSH connections by default.

HTTP Response Times for

smokeping I'm a big fan of the Smokeping project developed by Tobi Oetiker, and I use it to graph ICMP, DNS, and HTTP round trip times for my local internet connections and also for Since the beginning of 2010, I've switched around where was hosted from a system at a hosting provider in Canada to a system running on my home network (serviced by a Verizon FiOS connection). The drop in RTT's in mid-February was expected and quite large - going from an average of around 200ms down to about 20ms. Then, in June, I switched to a different Verizon DSL connection and moved the webserver system to a newer Verizon FiOS connection at a different location, and the RTT's went up a bit to about 50ms. Here is the graph: 400 days

fwsnort Available in Fedora Repositories

fwsnort on Fedora The fwsnort project is now available directly through the Fedora RPM repositories (for Fedora 12 and 13) thanks to Guillermo Gomez. The version that is currently bundled is fwsnort-1.0.6. Once fwsnort-1.2 is released, the implementation of large iptables rule sets that are derived from Snort rules will become a lot faster. This is because fwsnort is going to support the iptables-save format by integrating the complex rules built by fwsnort with any existing iptables policy that is instantiated in the kernel. This is made possible by interpreting the local policy and splicing in all of the fwsnort rules in the right places - each iptables chain is built from scratch upon an iptables-restore (including the built-in chains), so integrating with a running policy is not as easy as just adding each fwsnort rule into a set of custom chains. Compatibility with the iptables-save format has largely been completed with this patch in the fwsnort-1.2 development effort.

If you are running a Fedora 12 or 13 system, you can install fwsnort like so: # yum install fwsnort

Pure C Implementation of Single Packet Authorization

Pure C Implementation of Single Packet Authorization For the past several months, the pure-C implementation of Single Packet Authorization with fwknop has been in development, and the first release candidate of fwknop-2.0.0 is ready for download. Damien Stuart is the primary developer of this code, and I wish to thank him for all of his efforts in making this release possible. The original perl version of fwknop will continue to receive bug fixes as they are found, but the primary development effort is now on the lightweight and portable C code. The Trac interface for the perl code has moved to fwknop-perl. Any release that is at fwknop-2.0.0 or above will be the C implementation, and any release in the 1.9 series or before are in perl (and will remain available here). The perl and C versions are compatible with each other, so SPA packets created by the fwknop perl client can be properly decrypted and decoded by the new fwknopd C server, and similarly SPA packets built with the fwknop C client can be interpreted by the fwknopd perl server.

The heart of the new C code is the libfko library, which is used by both the fwknop client and the fwknopd server for SPA packet encryption/decryption and interpretation. The libfko library is portable to several platforms including Linux and Windows, and the standard GNU autoconf project is used for building fwknop from sources.

Given the compact nature of the C code, the fwknop-2.0.0rc1 release candidate is already known to function on OpenWRT, and Damien Stuart (the primary developer of the fwknop C code) has built packages for the OpenWRT Kamikaze and Backfire releases. This brings SPA functionality to embedded systems, and has been tested on a Linksys WRT54GS V2.0 router. The OpenWRT packages are available here.

There will be a few release candidates of fwknop-2.0.0 to ensure proper operation on a variety of platforms, and the emphasis will be on stability and ironing out any bugs. Also, fwknop-2.0.0 will initially support iptables firewalls on the servers side, with ipfw support coming soon after. There is one nice feature in fwknop-2.0.0rc1 that deserves a mention - the fwknop client now supports the notion of a ~/.fwknoprc file that allows a set of stanzas to be defined for remote SPA servers and customize the command line arguments used to access each one. Here is an example stanza: # Example for a destination server of to open access to
# SSH for an IP that is resolved externally
ACCESS tcp/22
ALLOW_IP resolve
Now, with the fwknop client, we reference this stanza with the -n command line argument like so: $ fwknop -n myssh -vvv
Resolved external IP (via as:
Enter encryption password:

FKO Field Values:

Random Value: 8308719569638051
Username: mbr
Timestamp: 1280030806
FKO Version: 1.9.12
Message Type: 1
Message String:,tcp/22
Nat Access: <NULL>
Server Auth: <NULL>
Client Timeout: 0
Digest Type: 3

Encoded Data: 8308719569638051:bWJy:1280030806:1.9.12:1:OTYuMjQ0LjE3MS4yNTMsdGNwLzIy


Final Packed/Encrypted/Encoded Data:


Generating SPA packet:
protocol: udp
port: 50023
send_spa_packet: bytes sent: 182
On the fwknopd server side, we see the following printed to the terminal in --foreground mode as the SPA packet is received and an iptables ACCEPT rule is added for SSH access and then removed after 30 seconds. Using Digest Cache: '/usr/var/run/fwknop/digest.cache' (entry count = 93)
PCAP filter is: udp port 50023
Starting fwknopd main event loop.

SPA Packet from IP: received.
SPA Packet: '87wQ3s171if9x8628P6k1oBrQEJEmS5c/+ECed5cfDmu8hGhC5esUCEVcI8IH5TdjWva44efebX+cfXUVPBU9c/6tJpOLwC8k/8sN87SgYkBK+m64aD4FnnndaeqsrIcO/oHcpXHk9S/8rqqypcumNtlXVZD/isS3Q729LMTa+2uI6TsGX9OQ3'

SPA Decode (res=0):
SPA Field Values:
Random Value: 8308719569638051
Username: mbr
Timestamp: 1280030806
FKO Version: 1.9.12
Message Type: 1
Message String:,tcp/22
Nat Access: <NULL>
Server Auth: <NULL>
Client Timeout: 0
Digest Type: 3
Encoded Data: 8308719569638051:bWJy:1280030806:1.9.12:1:OTYuMjQ0LjE3MS4yNTMsdGNwLzIy

Added Rule to FWKNOP_INPUT for, tcp/22 expires at 1280030886
RES=0, CMD_BUF: /sbin/iptables -t filter -L FWKNOP_INPUT --line-numbers -n 2>&1
RULES LIST: Chain FWKNOP_INPUT (1 references)
num target prot opt source destination
1 ACCEPT tcp -- tcp dpt:22 /* _exp_1280030886 */

Removed rule 1 from FWKNOP_INPUT with expire time of 1280030886.

Software Release - psad-2.1.7

psad-2.1.7 released The 2.1.7 release of psad is available for download. This release adds some relatively minor functionality to switch whois lookups to the destination IP whenever the source IP is part of a directly connected subnet. This change gives better reporting data in normal psad email alerts. The complete ChangeLog entries are displayed below. The psad-2.1.6 release was also made a few days ago, and fixes a bug that caused psad to die on some Ubuntu systems when using the Date::Calc Decode_Month() function, and it also added the --Override-config feature so that alternate configuration variables can be specified via config files besides psad.conf (implemented by Franck Joncourt).

  • (Dan A. Dickey) Added the ability to use the "ip" command from the iproute2 tools to acquire IP addresses from local interfaces. Dan's description is as follows: "...A main reason for doing this is in the case of multi-homed hosts. ifconfig sets these up on an interface using aliases, iproute2 does not. So, for a multi-homed interface (eth0 with multiple addresses), ifconfig -a only shows the first one configured and not the rest. ip addr shows all of the configured addresses...".
  • Added ENABLE_WHOIS_FORCE_ASCII to replace any non-ascii characters in whois data (which is common with whois lookups against Chinese IP addresses for example) with the string "NA". This option is disabled by default, but can be useful if errors like the following are seen upon receiving an email alert from psad:

    <<< 554 5.6.1 Eight bit data not allowed
    554 5.0.0 Service unavailable

  • Updated psad to issue whois lookups against IP addresses that are not directly connected to the local system. This is useful for example when an internal system is scanning an external destination system, and the scan is logged in the FORWARD chain. Issuing whois lookups on the internal system (frequently on RFC 1918 address space) is not usually very useful, but issuing the whois lookup against the destination system gives much more interesting data. This feature can be disabled with the new ENABLE_WHOIS_FORCE_SRC_IP variable.

Software Release - fwsnort-1.1

software release fwsnort-1.1 The 1.1 release of fwsnort is ready for download. This is a significant release that adds support for ip6tables so that SNORT ® inspection logic can be applied to IPv6 traffic within the Linux kernel. This release also includes a new feature that allows fwsnort to build perl commands interfaced with netcat that generate packet data that matches Snort rules (for those that that can be faithfully translated - see the --include-perl-triggers command line argument and associated comments within the file).

Here is the complete fwsnort-1.1 ChangeLog:

  • Added the ability to build an fwsnort policy that utilizes ip6tables instead of iptables. This allows fwsnort filtering and altering capabilities to apply to IPv6 traffic instead of just IPv4 traffic. To enable ip6tables usage, use the "-6" or "--ip6tables" command line arguments.
  • Added the --include-perl-triggers command line argument so that translated Snort rules can easily be tested. This argument instructs fwsnort to include 'perl -e print ... ' commands as comments in the /etc/fwsnort/ script, and these commands can be combined with netcat to send payloads across the wire that match Snort rules.
  • Updated fwsnort to create logs in the /var/log/fwsnort/ directory instead of directly in the /var/log/ directory. The path is controlled by a new variable 'LOG_FILE' in the /etc/fwsnort/fwsnort.conf file.
  • Added several variables in /etc/fwsnort/fwsnort.conf to control paths to everything from the config file to the snort rules path. Coupled with this is the ability to create variables within path components and fwsnort will expand them (e.g. 'CONF_DIR /etc/fwsnort; CONF_FILE $CONF_DIR/fwsnort.conf').
  • Added --Last-cmd arg so that it is easy to rebuild the script with the same command line args as the previous execution.
Snort is a registered trademark of Sourcefire, Inc. on SPA Ghost Services on fwknop Russ McRee of has written the January Toolsmith issue from the ISSA Journal about fwknop and the ability to create ghost services with Single Packet Authorization. In his Toolsmith paper, Russ emphasizes the possibility of using the ghost services concept to bypass strict outbound network filtering rules on a local network in order to access an external service that is bound to a port that is filtered by the local firewall. That is, the service is made accessible by having the SPA packet created by the fwknop client request that the remote fwknopd server create iptables DNAT rules to forward connections to a port that the local network actually allows out to the port where the service is bound. Russ uses this concept to access a file that is piped through a netcat listener on TCP port 6543, but do it from the heavily filtered network over TCP port 110 (normally associated with pop3).

Here is a link to the Toolsmith PDF entitled "Single Packet Authorization: The Ghost in the Machine".