Installation should be performed on two distinct machines with vanilla Debian testing (Wheezy), 64-bit and direct access to the Internet (this version does not support proxies yet). One machine (let’s call it hsn2-main) will hold HSN 2.0 components and the GUI, the other one (let’s call it hsn2-capture) will hold Capture Service and Server. You of course have to install Capture-HPC Server.
Steps for hsn2-main
First, you need to install proper Debian packages.
# apt-get install hsn2-framework hsn2-os hsn2-data-store hsn2-file-feeder hsn2-js-sta hsn2-norm-url hsn2-rb-archiveinflate hsn2-rb-clamavnugget hsn2-rb-officecat hsn2-rb-pdffox hsn2-rb-swfscanner hsn2-rb-virustotal hsn2-reporter hsn2-shell-scdbg hsn2-swf-cve hsn2-webclient hsn2-url-feeder hsn2-console libemu python-pika
HSN 2.0 is based on Java 7 and AMQP implementation called RabbitMQ so you must install it on the machine.
# apt-get install rabbitmq-server openjdk-7-jre
Steps for hsn2-capture
You need to install proper Debian packages.
# apt-get install hsn2-console hsn2-capture-hpc python-hsn2-lib python-hsn2-proto
Finally, you need to configure hc tool on the second machine. In order to do that you need to open /etc/hsn2/console.conf and change the server attribute in rabbitmq section.
Steps for hsn2-main
You need to edit /etc/hsn2/razorback/virustotal.conf file and include your VirusTotal API key in the following line.
API_KEY="YOUR API KEY GOES IN HERE";
We also supply an example workflow (configuration of URL processing) for testing. In order to use it you have to clone a GIT repository.
# git clone git://github.com/CERT-Polska/hsn2-workflows.git /etc/hsn2/workflows
Over time we will update this repository with additional workflows. Remember to go to your /etc/hsn2/workflows and do git pull from time to time.
Finally start all of the HSN2 services.
# service hsn2-framework start
# service hsn2-os start
# service hsn2-data-store start
# service hsn2-file-feeder start
# service hsn2-js-sta start
# service hsn2-norm-url start
# service hsn2-rb-archiveinflate start
# service hsn2-rb-clamavnugget start
# service hsn2-rb-officecat start
# service hsn2-rb-pdffox start
# service hsn2-rb-swfscanner start
# service hsn2-rb-virustotal start
# service hsn2-reporter start
# service hsn2-shell-scdbg start
# service hsn2-swf-cve start
# service hsn2-webclient start
# service hsn2-url-feeder start
Steps for hsn2-capture
You will also need to configure Capture-HPC service. Open /etc/init.d/hsn2-capture-hpc file and change connector and datastore addresses such that they point to hsn2-main. Next, start the service:
# service hsn2-capture-hpc start
WebGUI can be installed on any of the two previous machines or on any other machine. However, if you want to use a different machine, please remember to install hsn2-console, python-hsn2-lib and python-hsn2-proto packages and change the server attribute in rabbitmq section of /etc/hsn2/console.conf.
First you’ll need to install its dependencies.
# apt-get install curl python-dev libmysqlclient-dev python-django python-mysqldb python-pika python-crypto python-paramiko python-couchdb python-dateutil python-apscheduler
# apt-get install mysql-server apache2 libapache2-mod-wsgi
Next, you need to create database user and give her sufficient privileges.
# mysql -u root -p
mysql> CREATE USER 'hsn2'@'localhost' IDENTIFIED BY 'somepassword';
mysql> CREATE DATABASE hsn2;
mysql> GRANT ALL ON hsn2.* TO 'hsn2'@'localhost';
You also need to copy all the GUI files to an appropriate directory.
# git clone https://github.com/CERT-Polska/hsn2-webgui /var/www/hsn2
$ cd /var/www/hsn2/web
You also need to edit /var/www/hsn2/web/settings.py and specify MySQL password you entered previously (in line 18). Now you are ready to create WebGUI databases.
$ python manage.py syncdb
$ curl -X PUT http://localhost:5984/hsn
$ cd /var/www/hsn2/etc/couchdb_views
$ python couchdb_views.py -s
You also need to configure Apache Server and the backend. First, open the /var/www/hsn2/etc/scheduler.conf file. In the very last section of it (sftp) you must provide username/password combination that will be used to login to hsn2-main (localhost). Usually, you’ll need to create a new user for this and give her at least permissions to write to the remote_path directory. Then you need to do some further configuration.
# rm /var/www/index.html
# cat /var/www/hsn2/apache/hsn2.conf >> /etc/apache2/apache2.conf
# chmod ugo+w /var/www/hsn2/web/upload/
$ cd /var/www/hsn2/etc/
$ python synchronizer.py workflows
$ python scheduler.py start
Now all that’s left is to restart apache2 service and you are ready to use HSN 2.0.
# service apache2 restart
Submitting a job.
To submit a job you must go to "Create New Job" view. First, you select the workflow (for now there should be only one). Then, you need to select "Feeder file". It is a file with a list of URLs, each one in a separate line. All this URLs will be processed and (in case you are using default workflow) also links present on them will be processed. Finally, there are some self-explanatory options that you can tweak. After a successful submission, you should see a page with job details.
Viewing job results.
Job results can be viewed from the "Job Overview" page. When you click on a job name that you are interested in, you can see job details. Then, clicking on the job, you will see a list of top-level URLs and their classification. Then, by clicking on the URL you can view all objects associated with it (links, redirects, iframes etc.) and their classification. When you click on this object you will see a list of all services that were run against that object. Clicking on any of it will provide you with all information gathered by the selected service.
Basic console usage
HSN 2.0 can also be used from console tool (hc). Most useful commands are listed below. Command shortcuts are included in parentheses.
hc ping (hc p)
pinging framework to see if it responds (kind of sanity check).
hc job list (hc j l)
lists all jobs that were started between system start and present.
hc job details <job_id> (hc j d <job_id>)
lists all details about the specified job, based on its id.
hc workflow list (hc w l)
lists all workflows present in the repository.
hc workflow upload --file <path> <workflow_name> (hc w u -f <path> <workflow_name>)
upload new workflows to the repository. Workflow should be located at <path> and will be uploaded as <workflows_name>.
If any of the jobs is aborted and the reason for this behavior is unclear to you, you might have run into a bug. If you believe this is the case, please use the information below to report this situation back to us.
There are several known bugs in this release of HSN 2.0:
There are problems with GUI file upload, so sometimes you have to submit the file twice. This situation can be easily checked with console tool. Reason for failure should state that the file was not found.
File objects are different objects than URL objects. The situation is so by design. That means that if there is any file analyzed it will be shown twice in the results: once as a link and will be analyzed by webclient, and second time as a file. Both object will have the same URL.
Only way to stop the job is to restart the framework.