Untangle Network Security

Table of Contents

Untangle Network Security

Credits

About the Author

About the Reviewers

www.PacktPub.com

Support files, eBooks, discount offers, and more

Why subscribe?

Free access for Packt account holders

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Errata

Piracy

Questions

1. Introduction to Untangle

Introducing Untangle, Inc.

An overview on information security

The CIA triad

Types of attacks

Types of controls

Defense in depth

Introducing Untangle NGFW

Untangle NGFW modules

Untangle packages

Licensing Untangle

Reviewing the change log

Summary

2. Installing Untangle

Understanding the hardware requirements of Untangle NGFW

Untangle NGFW appliances

Building your Untangle NGFW box

Virtualizing your Untangle NGFW

Tweaking your Untangle NGFW

Setting up your lab

Getting Untangle

Writing your image

Untangle NGFW installation guide

Step 1 – booting and selecting the installation mode

Step 2 – selecting the installation wizard language

Step 3 – configuring the system locale

Step 4 – configuring the keymaps

Step 5 – configuring the server's time zone

Step 6 – reviewing the hardware rating summary

Step 7 – preparing the hard disk

Step 8 – completing the installation

Summary

3. The Initial Configuration of Untangle

Understanding the boot options

The initial configuration wizard

Step 1 – selecting the wizard language

Step 2 – setting the admin password and server's time zone

Step 3 – mapping the network cards

Step 4 – configuring the Internet connection

Acquiring automatic configurations from DHCP

Manually configuring the interface settings

Step 5 – configuring the Untangle NGFW operation mode

Understanding the router operation mode

Understanding the transparent bridge operation mode

Step 6 – configuring the automatic upgrade settings

Step 7 – finishing the initial configuration wizard

Registering your server

Reviewing the GUI

Untangle NGFW administration options

Summary

4. Untangle Advanced Configuration

Untangle placement options

Understanding the architecture of Untangle NGFW

Managing Untangle NGFW interfaces

Common uses of additional interfaces

Configuring Untangle NGFW interfaces

Addressed interfaces

Bridged interfaces

VLANs

Configuring Untangle NGFW high availability

Configuring the Untangle NGFW hostname

Configuring Untangle NGFW Services ports

Untangle NGFW network services

Untangle NGFW as a router

The Untangle NGFW DNS service

The Untangle NGFW DHCP service

DNS and DHCP advanced options

Configuring advanced network options

Understanding Untangle NGFW rules

Port forward rules

NAT rules

Bypass rules

QoS rules

The seven priorities

Configuring the QoS settings

Configuring the QoS rules

Reviewing the QoS status

Filter rules

Troubleshooting

Summary

5. Advanced Administration Settings

Configuring the administration settings

Managing the administrator accounts

Configuring the remote administration settings

Configuring the public address of Untangle NGFW

Configuring the regional settings

Configuring Untangle NGFW processing of protocols

Understating the available support settings

Changing Untangle NGFW skins

Managing Untangle SSL certificates

The certificate authority

The server certificate

Configuring the e-mail settings of Untangle NGFW

Configuring the outgoing e-mail server

Configuring trusted senders

Managing the Untangle NGFW quarantine

Accessing Untangle's quarantine web application

Managing the local directory of Untangle NGFW

Upgrading Untangle

Backing up and restoring

Backing up and restoring all Untangle NGFW configurations

Backing up and restoring individual settings

Monitoring your Untangle NGFW

Using SNMP

Syslog and summary reports

Reviewing system information and license details

Server information

The Licenses tab

License agreement

Summary

6. Untangle Blockers

Dealing with Untangle NGFW modules

Protect your network from viruses

How the antivirus programs work

Understanding the technical details of Untangle Virus Blocker

Virus Blocker settings

Configuring the scanning of the web traffic

Configuring the scanning of the SMTP traffic

Scanning FTP traffic settings

Reviewing the scan history

Identifying the common issues with Untangle Virus Blocker

Lab-based training

Testing web scanning

Testing e-mail scanning

Testing FTP scanning

Spam!!…It's something from the past

How anti-spam programs work

Understanding the technical details of Untangle Spam Blocker

Spam Blocker settings

Reviewing the scan history

The spam blocker event log

The tarpit event log

Reports

Common issues with Spam Blocker

Lab-based training

Testing the blocking of incoming spam

Testing the blocking of outgoing spam

Testing the marking of spam message functionality

Testing the quarantine functionality

Accessing the quarantine

Administrative management of users' quarantines

No more phishing

Technical details of Untangle Phish Blocker

Phish Blocker settings

Reviewing the scan history

Utilizing Untangle Ad Blocker

How it works

Understanding the settings of Untangle Ad Blocker

Status

Ad Filters

Cookie filters

Pass Lists

Reviewing the scan history

Lab-based training

Summary

7. Preventing External Attacks

Protecting against DoS attacks

Managing the shield

Reviewing the shield events

Lab-based training

Intrusion prevention using Untangle NGFW

How intrusion prevention systems work

IDS versus IPS

Identification methods

Counter measures

Technical details

Intrusion Prevention settings

Status

Reports

Reviewing the scan history

Lab-based training

Understanding Untangle's Firewall application

Technical details

Firewall settings

Reviewing the events of the Firewall application

Lab-based training

Summary

8. Untangle Filters

Untangle Web Filter

Working of Web Filter

Technical details

Block lists

Category-based website blocking

Blocking individual websites

Blocking certain files and MIME types

Allowing lists

HTTPS' advanced options

Other advanced options

Reviewing the history

Utilizing HTTPS Inspector

Untangle and HTTPS

Working of HTTPS Inspector

Configuring clients to trust Untangle's root CA

The manual method

Deploying the root CA certificate using GPO

Configuring HTTPS Inspector

Reviewing the inspect activity

Untangle Application Control

Untangle Application Control Lite

Adding Application Control Lite signatures

Application Control Lite Status

Blocking applications/protocols

The paid version of Application Control

The Application Control status

Blocking applications/protocols

Reviewing the scanning history

Lab-based training

Configuring Web Filter settings

Testing the functionality of Web Filter

Configuring HTTPS Inspector settings

Testing the functionality of Web Filter

Configuring and testing Application Control settings

The cat and mouse game

Summary

9. Optimizing Network Traffic

Bandwidth Control

How does Bandwidth Control work?

Settings

Bandwidth Control rules

Bandwidth Control setup wizard

Rules

Bandwidth Monitor

The penalty box

Quotas

Reviewing the scan history

Lab-based training

Web Cache

Web Cache settings

Status

Cache Bypass

Reviewing the caching history

Lab-based training

Summary

10. Untangle Network Policy

Directory Connector

The User Notification API

UNLS

The Active Directory Login Monitor Agent

Configuring Active Directory Connector

Connecting Untangle to a RADIUS server

Directory Connector reports

Untangle's Captive Portal

The working of Captive Portal

Configuring Captive Portal

Setting traffic capture rules

Common traffic capture rules

Configuring the passed hosts

Customizing the captive page

Setting the user authentication method

Reviewing Captive Portal events

Untangle's Policy Manager

Configuring Policy Manager policies

Parent and child racks

Configuring Policy Manager rules

Reviewing the Policy Manager events

Summary

11. Untangle WAN Services

WAN Failover

Setting up interface tests

Reviewing the WAN Failover events

WAN Balancer

Configuring traffic allocation

Setting Route Rules

Reviewing the WAN Balancer status

Troubleshooting

Summary

12. Untangle VPN Services

Understanding VPN

OpenVPN

How OpenVPN works

Configuring Untangle's OpenVPN server settings

Configuring Untangle's OpenVPN remote client settings

Creating a remote client

Understanding remote client groups

Defining the exported networks

Using OpenVPN remote access clients

Using an OpenVPN client with Windows OS

Using OpenVPN with non-Windows clients

Using OpenVPN for site-to-site connection

Reviewing the connection details

Troubleshooting Untangle's OpenVPN

Lab-based training

IPsec VPN

How the IPsec VPN works

Configuring Untangle's IPsec VPN

Creating IPsec tunnels

Configuring L2TP options

Reviewing the connection events

Lab-based training

Summary

13. Untangle Administrative Services

Untangle's Reports

Configuring the settings of Untangle's Reports

Viewing Untangle's Reports

Branding Manager

Live Support

Configuration backup

Summary

14. Untangle in the Real World

Understanding the IT regulatory compliance

Untangle in real life

Untangle's advantages

Untangle for SMB

Using Untangle in education

Using Untangle in healthcare

Using Untangle in government organizations

Using Untangle in nonprofit organizations

Summary

Index

Untangle Network Security

Untangle Network Security

Copyright © 2014 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: October 2014

Production reference: 1251014

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-84951-772-0

www.packtpub.com

Cover image by Pratik P Prabhu (<pratikpprabhu@gmail.com>)

Credits

Author

Abd El-Monem A. El-Bawab

Reviewers

Ritwik Ghoshal

Vishrut Mehta

Gilbert Ramirez

Abhinav Singh

Tom Stephens

Acquisition Editor

Vinay Argekar

Content Development Editor

Athira Laji

Technical Editors

Faisal Siddiqui

Ankita Thakur

Copy Editors

Janbal Dharmaraj

Alfida Paiva

Project Coordinator

Harshal Ved

Proofreaders

Simran Bhogal

Maria Gould

Ameesha Green

Paul Hindle

Indexer

Monica Ajmera Mehta

Graphics

Abhinash Sahu

Production Coordinators

Arvindkumar Gupta

Conidon Miranda

Cover Work

Conidon Miranda

About the Author

Abd El-Monem A. El-Bawab is a systems engineer with a passion for security. He has about 3 years of experience in the IT field. He is MCITP 2008 Server Administrator, MCSA 2012, MCSE Server Infrastructure, MCSE Private Cloud, and ITIL certified.

He has considerable experience in Untangle's Firewall, TMG, McAfee Sidewinder, Trend Micro Worry-Free Business Security Services, Symantec Endpoint Protection, Symantec Backup Exec, Hyper-V, System Center Suite, ESXi, Citrix XenServer, VDI, Windows Servers, Active Directory, Exchange Server, Office 365, and SMART Service Desk.

You can follow him on Twitter at @Eng_Monem and visit his blog at amagsmb.wordpress.com.

I would like to thank my mother, brothers, sisters, and all my family members for their continuous support, encouragement, and understanding. Without their support, I wouldn't be able to produce this work.

Special thanks to Mahmoud Magdy for his encouragement and friendship. I would also like to thank Khaled Eldosuky and Ahmed Abou Zaid for their efforts to increase the technical content in the Arab world.

Also, I would like to thank everyone who has contributed to the publication of this book, including the publisher, technical reviewers, and editors.

About the Reviewers

Ritwik Ghoshal is a senior security analyst at Oracle Corporation. He is responsible for Oracle's software and hardware security assurance. His primary work areas are operating systems and desktop virtualization along with developing vulnerability management and tracking tools. Before joining Oracle in 2010, when the company acquired Sun Microsystems, he had been working with Sun since 2008 as part of Sun's Security Engineering team and Solaris team. At Oracle, he continues to be responsible for all Sun systems' products and Oracle's Linux and virtualization products.

He earned a Bachelor's degree in Computer Science and Engineering in 2008 from Heritage Institute of Technology, Kolkata, India.

I'm heavily indebted to my parents and Sara E Taverner for their continuous help and support.

Vishrut Mehta is currently in the fourth year at IIIT Hyderabad. He is doing his research in cloud computing and software-defined networks under the guidance of Dr. Vasudeva Varma and Dr. Reddy Raja. He has done a research internship at INRIA, France, in which he had to work on various challenges in multicloud systems. He also loves open source and has participated in Google Summer of Code 2013 while working on a project for Sahana Software Foundation. He was also involved in various start-ups and worked on some of the leading technologies.

He was the technical reviewer of Python Network Programming Cookbook, Packt Publishing.

I would like to thank my advisor Dr. Vasudeva Varma and Dr. Reddy Raja for guiding me and helping me in times of need.

Gilbert Ramirez develops software to help other programmers and developers get their job done. He has been a long-time contributor to Wireshark, the premier open source packet analyzer. At Cisco Systems, Inc., he is responsible for software build systems, workflow automation for engineers, and virtualization tools.

He has reviewed Network Analysis Using Wireshark Cookbook, Packt Publishing. He has also written books on Wireshark, including Wireshark & Ethereal Network Protocol Analyzer Toolkit; Nessus, Snort, & Ethereal Power Tools; and Ethereal Packet Sniffing, all published by Syngress Publishing, Inc,.

Abhinav Singh is a young information security specialist from India. He has keen interest in the field of Information Security and has adopted it as his full-time profession. His core work areas include malware analysis, network security, and systems and enterprise security. He is also the author of Metasploit Penetration Testing Cookbook Second Edition and Instant Wireshark, published by Packt Publishing.

Abhinav's work has been quoted in several InfoSec magazines and portals. He shares his day-to-day security encounters at www.securitycalculus.com.

Currently, he is working as a cybersecurity engineer for J.P. Morgan. You can follow him on Twitter at @abhinavbom. You can also contact him at <abhinavbom@gmail.com>.

Tom Stephens is passionate about software. He has worked on everything from web design to low-level systems engineering to quality management. His broad experience and adaptability has helped him gain a keen insight into software and technology as a whole.

www.PacktPub.com

Support files, eBooks, discount offers, and more

You might want to visit www.PacktPub.com for support files and downloads related to your book.

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.

http://PacktLib.PacktPub.com

Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can access, read and search across Packt's entire library of books.

Why subscribe?

Fully searchable across every book published by Packt

Copy and paste, print and bookmark content

On demand and accessible via web browser

Free access for Packt account holders

If you have an account with Packt at www.PacktPub.com, you can use this to access PacktLib today and view nine entirely free books. Simply use your login credentials for immediate access.

Preface

Nowadays, network security has become the trending topic besides cloud computing and virtualization. With the increasing number of cybercrimes, all networks, irrespective of whether they belong to a small or enterprise organization, needs to be protected.

Untangle NGFW provides a comprehensive platform that is built on the best-on-the-market application such as zVelo's web filtering technologies that are used by most UTM manufactures. Untangle provides a complete stack of applications that cover the needs of most users.

Accompanied with ease of use, high reliability, great support, and low prices, Untangle NGFW is considered a great choice for network protection.

This book is based on version 10.2.1; slight differences can be found between the different Untangle NGFW versions. Since you have the basic concepts provided here, you can easily deal with any change you meet. Although you can use the free-to-roam style to read this book, this is not advised for beginners since the advanced chapters have some dependencies on the earlier chapters.

What this book covers

Chapter 1, Introduction to Untangle, introduces you to the world of information security and Untangle. This book starts by giving a brief introduction about Untangle, the company. Then, it provides some information about security concepts. After that, it gives a detailed introduction to Untangle NGFW.

Chapter 2, Installing Untangle, guides you on how to build and install the Untangle NGFW server. This chapter first discusses the hardware requirements of Untangle NGFW. Then, it describes the virtualized environment used for this book's examples. Next, the chapter covers how to obtain Untangle installation media, and then it guides you through a step-by-step installation of Untangle NGFW.

Chapter 3, The Initial Configuration of Untangle, walks you through the initial configuration wizard of Untangle NGFW in which we configure the administrator account, interfaces' IPs, and Untangle mode. In addition, it explains the GUI of Untangle NGFW.

Chapter 4, Untangle Advanced Configuration, covers how to configure network-related settings such as interface IP, VLAN, DHCP, DNS, QoS, Routes, NAT, and port forwarding. Also, it discusses Untangle's high availability options.

Chapter 5, Advanced Administration Settings, covers the settings related to Untangle NGFW administration such as the administrators' accounts, Untangle public address, backup and restore, and the e-mail settings for Untangle NGFW to send e-mails to users.

Chapter 6, Untangle Blockers, covers the Untangle applications that protect your network from direct threats such as viruses, spam, phishing, and malicious traffic.

Chapter 7, Preventing External Attacks, covers how you can protect your network from the Denial of Service (DoS) attacks by using intrusion prevention systems to stop malicious traffic and using firewall to limit the number of opened ports.

Chapter 8, Untangle Filters, covers Untangle applications that improve the user's productivity and network performance by blocking access to least important sites such as social networks, and denying traffic from applications such as BitTorrent. In addition, this chapter covers how Untangle NGFW can scan and filter the HTTPS traffic.

Chapter 9, Optimizing Network Traffic, covers how you can save and optimize your WAN bandwidth by limiting nonbusiness-related applications and prioritizing business-related applications. In addition, this chapter will also cover the use of Web Cache to enhance users' browsing experience.

Chapter 10, Untangle Network Policy, shows how it's possible to set access rules based on the Active Directory user and group membership, and how to force users to accept the acceptable use policy before they start using your network resource.

Chapter 11, Untangle WAN Services, describes the Untangle NGFW modules that allow for using WAN services from multiple ISPs to provide higher throughput and a continuous WAN connection to users even if any of the connection has failed.

Chapter 12, Untangle VPN Services, covers the modules that allow Untangle NGFW to provide a VPN connection to its remote users or between two branches.

Chapter 13, Untangle Administrative Services, shows how administrators can simplify their tasks using reporting, automated backups, and premium support, and how they can customize Untangle logos, interfaces, and pages.

Chapter 14, Untangle in the Real World, provides a brief overview of the regulatory compliance related to the IT field. Then, it lists the Untangle NGFW advantages over its rivals. Finally, it provides some examples on how Untangle is used in the small and medium businesses, education, healthcare, government, and nonprofit sectors.

What you need for this book

The examples presented in this book assume that you have a computer system with enough RAM, hard drive space, and processing power to run a virtualized testing environment. Some examples will require the use of multiple virtual machines that run simultaneously. The complete virtualized environment used for this book is described in Chapter 2, Installing Untangle.

Who this book is for

This book is for anyone who wants to learn how to install, deploy, configure, and administrate Untangle NGFW. This book has been written for readers at a beginner's level either on network security or Untangle NGFW, but they should be familiar with networks. For those who have more experience with network security, this book can be their quick guide to learn Untangle NGFW. For those with experience on Untangle NGFW, this book can serve as a refresher and validation of their skills.

Conventions

In this book, you will find a number of styles of text that distinguish between different kinds of information. Here are some examples of these styles, and an explanation of their meaning.

Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows: Backups will store the current Untangle NGFW server settings to the .backup file.

New terms and important words are shown in bold. Words that you see on the screen, in menus or dialog boxes for example, appear in the text like this: If your country is not listed in the preceding list, you can choose other.

Note

Warnings or important notes appear in a box like this.

Tip

Tips and tricks appear like this.

Reader feedback

Feedback from our readers is always welcome. Let us know what you think about this book—what you liked or may have disliked. Reader feedback is important for us to develop titles that you really get the most out of.

To send us general feedback, simply send an e-mail to <feedback@packtpub.com>, and mention the book title via the subject of your message.

If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide on www.packtpub.com/authors.

Customer support

Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.

Errata

Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books—maybe a mistake in the text or the code—we would be grateful if you would report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on the errata submission form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded on our website, or added to any list of existing errata, under the Errata section of that title. Any existing errata can be viewed by selecting your title from http://www.packtpub.com/support.

Piracy

Piracy of copyright material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works, in any form, on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy.

Please contact us at <copyright@packtpub.com> with a link to the suspected pirated material.

We appreciate your help in protecting our authors, and our ability to bring you valuable content.

Questions

You can contact us at <questions@packtpub.com> if you are having a problem with any aspect of the book, and we will do our best to address it.

Chapter 1. Introduction to Untangle

This chapter will introduce you to the Untangle company and its products. Untangle has two product lines: Untangle NGFW and IC Control. In this chapter, we will introduce you to Untangle NGFW and the modules available to be installed on the NGFW.

This chapter will also cover some of the information security basics required to understand the importance of using Untangle NGFW to protect our networks. In addition, the major changes from version 9.4.2 to version 10.2.1 will be covered.

In this chapter, we will cover the following topics:

Introducing Untangle, Inc.

An overview of information security

Introducing Untangle NGFW

Reviewing the change log

Introducing Untangle, Inc.

Untangle was founded in 2003 as Metavize, Inc. by John Irwin and Dirk Morris with the vision of untangling the complexities of network security and control. In 2006, and after a venture funding round from CMEA Ventures and Rustic Canyon Partners, the company was renamed to Untangle, Inc. and named Bob Walters as the CEO.

Untangle's first product, and its most popular one, is the Untangle gateway platform, which is available under the GNU General Public License (GNU GPL) v2 license. The Untangle gateway platform is the world's first commercial-grade open source solution for blocking spam, spyware, viruses, adware, and unwanted content on the network. In 2014, after releasing their second product, Untangle, Inc. renamed the Untangle gateway platform to Untangle Next generation firewall (NGFW). Untangle NGFW is available as an appliance or as software to be installed on a dedicated device. The demo of Untangle NGFW is available at http://demo.untangle.com/.

In 2014, Untangle released its second product under the name of Internet Content (IC) Control. IC Control is an enterprise-grade solution to maximize Internet performance by allowing granular control for every traffic type, scaling to 10 Gbps and offering centralized management for multi-appliance, multi-domain deployments. IC Control is based on Cymphonix Corp. products, which is now part of Untangle, Inc. after Untangle, Inc. acquired it in October 2013. IC Control is now available as appliance only; however, Untangle, Inc. has the intension to convert it to a software-based solution as is the case with Untangle NGFW. The IC Control demo is available at http://icc-demo.untangle.com/.

Untangle, Inc. has over 400,000 customers, protecting nearly 5 million people, their computers, and networks. The main sectors that use Untangle products are education, healthcare, nonprofit, and state and local government.

An overview on information security

If you have a public IP, you and your company may be the next victim of the cybercrime business. 75 percent of Internet traffic is malicious (https://wiki.cac.washington.edu/download/attachments/7479159/White_Paper_6-Feb26-round2-AS-BE+DRAFT.doc) and the cybercrime business value equals 105 USD billion, which surpasses the value of the illegal drug trade worldwide. In addition, most of the cybercrime attacks are determined, not just opportunistic,