You should check installation instructions contained in INSTALL file if you are installing BruteForceBlocker v1.2 and later. Basically, you will need to proceed these steps:
You will need to add a new table to the pf config file. You can do so by adding lines to the pf.conf similar to these:
table <bruteforce> persist file “/path/to/tables/file”
block in log quick proto tcp from <bruteforce> to any port ssh
You will also need to add another auth line in /etc/syslog.conf similar to this one (note that you should keep your original one, since the BruteForceBlocker no longer logs to the auth.log file, instead it logs throught syslog module):
auth.info;authpriv.info | exec /path/to/bruteforceblocker.pl
You should also consider starting syslogd with -c option.