UNIT 1: Introduction to Penetration Testing and Reconnaissance

Goal-based penetration testing: Introduction to Penetration Testing, Different types of

threat actors, Conceptual overview of security testing, Common pitfalls of vulnerability
assessments, penetration testing, and red team exercises, Objective-based penetration testing,
The testing methodology Kali Linux & Red Team Tactics, Using CloudGoat and Faraday.

Open-source Intelligence and Reconnaissance: Basic Principles of Reconnaissance,

Scraping, Google Hacking Database, creating custom wordlist for cracking password.

Active Reconnaissance of External and Internal Networks: Stealth scanning techniques,

DNS reconnaissance, and route mapping, Employing comprehensive reconnaissance
applications, Identifying the external network infrastructure, Mapping beyond the firewall,
IDS/IPS identification, Enumerating hosts, port, operating system, and service discovery,
Writing your port scanner using netcat, Large-scale scanning, Machine Learning for
Reconnaissance.

Introductions:

What is Cyber Security?

Cyber security is the practice of defending computers, servers, mobile devices, electronic systems, networks,
and data from malicious attacks. It's also known as information technology security or electronic
information security. The term applies in a variety of contexts, from business to mobile computing, and can
be divided into a few common categories.
·        Network security is the practice of securing a computer network from intruders, whether targeted
attackers or opportunistic malware.
·        Application security focuses on keeping software and devices free of threats. A compromised application
could provide access to the data its designed to protect. Successful security begins in the design stage, well
before a program or device is deployed.
·        Information security protects the integrity and privacy of data, both in storage and in transit.
·        Operational security includes the processes and decisions for handling and protecting data assets. The
permissions users have when accessing a network and the procedures that determine how and where data
may be stored or shared all fall under this umbrella.
·        Disaster recovery and business continuity define how an organization responds to a cyber-security
incident or any other event that causes the loss of operations or data. Disaster recovery policies dictate how
the organization restores its operations and information to return to the same operating capacity as before the
event. Business continuity is the plan the organization falls back on while trying to operate without certain
resources.
·        End-user education addresses the most unpredictable cyber-security factor: people. Anyone can
accidentally introduce a virus to an otherwise secure system by failing to follow good security practices.
Teaching users to delete suspicious email attachments, not plug in unidentified USB drives, and various
other important lessons is vital for the security of any organization.

Types of Cyber Attacks:

A cyber-attack is a deliberate attempt by external or internal threats or attackers to exploit and compromise
the integrity and confidentiality of the information system of a target organization.
Cyber-attacks come in a wide variety, and the following list highlights some of the important ones that
criminals and attackers use to exploit software:
Types of Hackers:
A Hacker is a person who is intensely interested in the mysterious workings of any computer operating
system. Hackers are most often programmers. They gather advanced knowledge of operating systems and
programming languages and discover loopholes within systems and the reasons for such loopholes.

There are generally 10-types of Hackers, they are:

White Hat Hackers: White hat hackers are the one who is authorized or the certified hackers who work for
the government and organizations by performing penetration testing and identifying loopholes in their
cybersecurity. They also ensure the protection from the malicious cyber crimes. They work under the rules
and regulations provided by the government, that’s why they are called Ethical hackers or Cybersecurity
experts.

Black Hat Hackers: They are often called Crackers. Black Hat Hackers can gain the unauthorized access of
your system and destroy your vital data. The method of attacking they use common hacking practices they
have learned earlier. They are considered to be as criminals and can be easily identified because of their
malicious actions.

Gray Hat Hackers: Gray hat hackers fall somewhere in the category between white hat and black hat
hackers. They are not legally authorized hackers. They work with both good and bad intentions; they can use
their skills for personal gain. It all depends upon the hacker. If a gray hat hacker uses his skill for his
personal gains, he/she is considered as black hat hackers.

Script Kiddies: They are the most dangerous people in terms of hackers. A Script kiddie is an unskilled
person who uses scripts or downloads tools available for hacking provided by other hackers. They attempt to
attack computer systems and networks and deface websites. Their main purpose is to impress their friends
and society. Generally, Script Kiddies are juveniles who are unskilled about hacking.

Green Hat Hackers: They are also amateurs in the world of hacking but they are bit different from script
kiddies. They care about hacking and strive to become full-blown hackers. They are inspired by the hackers
and ask them few questions about. While hackers are answering their question they will listen to its novelty.

Blue Hat Hackers: They are much like the script kiddies; are beginners in the field of hacking. If anyone
makes angry a script kiddie and he/she may take revenge, then they are considered as the blue hat hackers.
Blue Hat hackers payback to those who have challenged them or angry them. Like the Script Kiddies, Blue
hat hackers also have no desire to learn.

Red Hat Hackers: They are also known as the eagle-eyed hackers. Like white hat hackers, red hat hackers
also aims to halt the black hat hackers. There is a major difference in the way they operate. They become
ruthless while dealing with malware actions of the black hat hackers. Red hat hacker will keep on attacking
the hacker aggressively that the hacker may know it as well have to replace the whole system.

State/Nation Sponsored Hackers: State or Nation sponsored hackers are those who are appointed by the
government to provide them cybersecurity and to gain confidential information from other countries to stay
at the top or to avoid any kind of danger to the country. They are highly paid government workers.

Hacktivist: These are also called the online versions of the activists. Hacktivist is a hacker or a group of
anonymous hackers who gain unauthorized access to government’s computer files and networks for further
social or political ends.

Malicious Insider or Whistleblower: A malicious insider or a whistleblower could be an employee of a

company or a government agency with a grudge or a strategic employee who becomes aware of any illegal
activities happening within the organization and can blackmail the organization for his/her personal gain.

Introduction to Penetration Testing

What is Penetration Testing?

Penetration testing is a type of security testing that is used to test the insecurity of an application. It is
conducted to find the security risk which might be present in the system.

If a system is not secured, then any attacker can disrupt or take authorized access to that system. Security
risk is normally an accidental error that occurs while developing and implementing the software. For
example, configuration errors, design errors, and software bugs, etc.

Why is Penetration Testing Required?

Penetration testing normally evaluates a system’s ability to protect its networks, applications, endpoints and
users from external or internal threats. It also attempts to protect the security controls and ensures only
authorized access.

Penetration testing is essential because −

It identifies a simulation environment i.e., how an intruder may attack the system through white hat attack.

It helps to find weak areas where an intruder can attack to gain access to the computer’s features and data.

It supports to avoid black hat attack and protects the original data.

It estimates the magnitude of the attack on potential business.

It provides evidence to suggest, why it is important to increase investments in security aspect of technology

When to Perform Penetration Testing?

Penetration testing is an essential feature that needs to be performed regularly for securing the functioning of
a system. In addition to this, it should be performed whenever −

Security system discovers new threats by attackers.

You add a new network infrastructure.
You update your system or install new software.
You relocate your office.
You set up a new end-user program/policy.
How is Penetration Testing Beneficial?
Penetration testing offers the following benefits −
Enhancement of the Management System − It provides detailed information about the security threats. In
addition to this, it also categorizes the degree of vulnerabilities and suggests you, which one is more
vulnerable and which one is less. So, you can easily and accurately manage your security system by
allocating the security resources accordingly.

Avoid Fines − Penetration testing keeps your organization’s major activities updated and complies with the
auditing system. So, penetration testing protects you from giving fines.

Protection from Financial Damage − A simple breach of security system may cause millions of dollars of
damage. Penetration testing can protect your organization from such damages.

Customer Protection − Breach of even a single customer’s data may cause big financial damage as well as
reputation damage. It protects the organizations who deal with the customers and keep their data intact.

Different types of threat actors:

Different types of threat actors
A threat actor is nothing but an entity or individual who is responsible for an event or incident
that impacts another entity. It is important that we understand the different types of threat actors
and their common motivations, which will help us throughout this book to understand different
perspectives. Table 1.1 provides the common threat actors, their motives, and typical goals.

Threat Actor Common Motivation Goal(s)

State- or governmentsponsored Military, political, and Cyber espionage, data theft, or
actors technological agendas any other activity that interests a
nation for its economic benefits
Organized crime or Financial gain and profit Money and valuable data
cybercriminals
Hacktivists/cyber extremists Motivational overlaps Focus on exposing secrets and
disrupting services/organizations
that they think are not good for
society (hacktivists); focus on
causing harm and destruction to
further their cause (extremists)
Insiders Revenge Money or data ransom or creating
revenue loss

Common pitfalls of vulnerability assessments, penetration testing, and red team exercises

• Vulnerability Assessment (VA): The process of identifying vulnerabilities or security loopholes in a

system or network through a vulnerability scanner. One of the misconceptions about VA is that it will let
you find all of the known vulnerabilities; well, that’s not true. Limitations with VA include that only
potential vulnerabilities are found, and it depends purely on the type of scanner that you utilize. It might also
include a number of false positives and, to the business owner, there is no clear indication as to which ones
do not pose a relevant risk and which one will be initially utilized by the attackers to gain access. The
biggest pitfall of VA is false negatives, meaning the scanner did not find an issue that the system or
application has.

• Penetration testing (pentesting): The process of safely simulating the hacking scenarios by exploiting
vulnerabilities without much impact on the existing network or business. There is also a lower number of
false positives since testers will try to validate the vulnerabilities and also attempt to exploit them. A
limitation with pentesting is that it uses only currently known, publicly available exploits; mostly, these are a
focus for project testing. We often hear from pentesters during an assessment, Yay! Got Root—but we never
hear the question, what can you do with it? This could be due to various reasons such as project limitations,
including the reporting of high-risk issues immediately to the client, or the client only being interested in one
segment of the network and only wanting that part tested.

• Red Team Exercise (RTE): A focused process of evaluating the effectiveness of an organization to defend
against cyber threats and improve its security by any possible means; during an RTE, we can discover
multiple ways of achieving project objectives/ scenarios and goals, such as complete coverage of activities
with the defined project goal, including phishing (enticing a victim to enter sensitive information or
download malicious content through emails), vishing (enticing a victim to provide or do some actions with
malicious intent through phone calls), “WhatsApping” (engaging a victim through WhatsApp messenger
with malicious intent), wireless, disk drops (USB and SSD), and physical penetration testing. The limitations
with RTEs are time-bound, pre-defined scenarios and an assumed rather than real environment. Often, the
RTE is run with a fully monitored mode for every technique, and tactics are executed according to the
procedure, but this isn’t the case when a real attacker wants to achieve an objective.

The following table illustrates the fundamental differences between penetration testing and vulnerability
assessments −
Penetration Testing Vulnerability Assessments

Makes a directory of assets and resources in a given system.

Determines the scope of an attack.

Discovers the potential threats to each resource.

Tests sensitive data collection.

Gathers targeted information and/or inspect the system. Allocates quantifiable value and significance to the available
resources.

Attempts to mitigate or eliminate the potential vulnerabilities of

Cleans up the system and gives final report. valuable resources.

It is non-intrusive, documentation and environmental Comprehensive analysis and through review of the target system
review and analysis. and its environment.

It is ideal for physical environments and network

architecture. It is ideal for lab environments.

It is meant for critical real-time systems. It is meant for non-critical systems.

Objective-based penetration testing

Objective-based penetration testing The primary goal of a pentest/RTE is to determine the real risk,
differentiating the risk rating from the scanner and giving a business a risk value for each asset, along with
the risk to the brand image of the organization. It’s not about how much risk they have; rather, how much
they are exposed and how easy it is to exploit this exposure. A threat that has been found does not really
constitute a risk and need not be demonstrated; for example, Cross-Site Scripting (XSS) is a script injection
vulnerability that can steal users’ credentials. If a client running a trading company had a brochure website
that provides static content to their customers was vulnerable to XSS, it may not have a significant impact
on the business. In this case, a client might accept the risk and put in a mitigation plan using a Web
Application Firewall (WAF) to prevent the XSS attacks. If the same vulnerability was identified on their
main trading website, however, then it would be a significant issue in need of rectification as soon as
possible since the company will be at risk of losing the trust of customers through attackers stealing their
credentials. Objective-based penetration testing is time-based, depending on the specific problem that an
organization faces. An example of an objective is: We are most worried about our data being stolen and the
regulatory fines incurred as a consequence of these breaches. So, the objective now is to compromise the
data either by exploiting a system flaw or by manipulating the employees through phishing; sometimes it
will be a surprise to see some of their data is already available on the dark web. Every objective comes with
its own Tactics, Techniques, and Procedures (TTP) that will support the primary goal of the penetration test
activity. We will be exploring all of these different methodologies throughout this book using Kali Linux
2021.4.

Basic principles of reconnaissance

Reconnaissance, or recon, is the first step of the kill chain when conducting a penetration test or attack
against a data target. This is conducted before the actual test or attack of a target network. The findings will
give a direction to where additional reconnaissance may be required, or the vulnerabilities to attack during
the exploitation phase.
Reconnaissance activities are segmented on a gradient of interactivity with the target network or device.

Passive reconnaissance does not involve any malicious direct interaction with the target network The
attacker's source IP address and activities are not logged (for example, a Google search for the target's email
addresses). It is difficult, if not impossible, for the target to differentiate passive reconnaissance from normal
business activities.

• Passive reconnaissance is further divided into Direct and Indirect categories, while direct passive
reconnaissance involves the normal interactions that occur when an attacker interacts with the target in an
expected manner. For example, an attacker will log on to the corporate website, view various pages, and
download documents for further study. These interactions are expected user activities, and are rarely
detected as a prelude to an attack on the target. Whilst the Indirect passive reconnaissance there will be
absolutely no interaction with the target organisation

• Active reconnaissance involves direct queries or other interactions (for example, port scanning of the
target network) that can trigger system alarms or allow the target to capture the attacker's IP address and
activities. This information could be used to identify and arrest an attacker, or during legal proceedings.
Because active reconnaissance requires additional techniques for the tester to remain undetected, it will be
covered in Chapter Active Reconnaissance of External and Internal Networks.

Penetration testers or attackers generally follow a process of structured information gathering, moving from
a broad scope (the business and regulatory environments) to the very specific (user account data).

To be effective, testers should know exactly what they are looking for and how the data will be used before
collection starts. Using passive reconnaissance and limiting the amount of data collected minimizes the risks
of being detected by the target.

Open-Source Intelligence and Passive Reconnaissance

Active reconnaissance is the art of collecting information directly from a target. The purpose of this phase is
to collect and weaponize information about the target to the greatest degree possible to facilitate the
exploitation phase of the kill chain methodology. We saw in the last chapter how to perform passive
reconnaissance using OSINT, which is almost undetectable and can yield a significant amount of
information about the target organization and its users. This phase builds on the results obtained from
OSINT and passive reconnaissance and emphasizes more focused probing to identify the path to, and the
attack surface of, a target. In general, complex systems have a greater attack surface, and each surface may
be exploited and then leveraged to support additional attacks. Although active reconnaissance produces
more useful information, interactions with the target system may be logged, triggering alarms by protective
devices, such as firewalls, Intrusion Detection Systems (IDSes), Intrusion Prevention Systems (IPSes), and
Endpoint Detection Response (EDR) systems.

As the usefulness of the data to the attacker increases, so does the risk of detection; this is shown in Figure
3.1
To improve the effectiveness of active reconnaissance in providing detailed information, our
focus will be on using the stealthiest techniques, as these will be the most difficult to detect. In
this chapter, you will learn about the following:
• Stealth scanning techniques
• External and internal infrastructure, host discovery, and enumeration
• Comprehensive reconnaissance of applications, especially recon-ng
• Enumeration of internal hosts using DHCP
• Enumerating services within the SaaS applications
• Useful Microsoft Windows commands during penetration testing
• Taking advantage of default configurations
• Enumeration of users using SNMP, SMB, and rpcclient

Stealth scanning techniques

The greatest risk of active reconnaissance is discovery by a target. Using the tester’s time and data stamps,
the source IP address, and additional information, the target can identify the source of the incoming
reconnaissance.
Therefore, stealth techniques are employed to minimize the chances of detection. When employing
stealth to support reconnaissance, a tester mimicking the actions of a hacker will do the following:
• Camouflage tool signatures to avoid detection and thereby trigger an alarm
• Hide the attack within legitimate traffic
• Modify the attack to hide the source and type of traffic
• Make the attack invisible using non-standard traffic types or encryption
Stealth scanning techniques can include some or all of the following:
• Adjusting source IP stack and tool identification settings
• Modifying packet parameters (Nmap)
• Using proxies with anonymity networks (ProxyChains and the Tor network)

DNS reconnaissance and route mapping

Once a tester has identified the targets that have an online presence and contain items of interest, the next
step is to identify the IP addresses and routes to the target. DNS reconnaissance is concerned with
identifying who owns a particular domain or series of IP addresses (the sort of information gained with
whois, although this has been completely changed with the General Data Protection Regulation (GDPR)
enforcement across Europe from May 2018). The DNS information defines the actual domain names and IP
addresses assigned to the target, and the route between the penetration tester—or the attacker—and the final
target. This information gathering is semi-active, as some of the information is available from freely
available open sources such as dnsdumpster.com, while other information is available from third parties
such as DNS registrars. Although the registrar may collect IP addresses and data concerning requests made
by the attacker, it is rarely provided to the end target. The information that could be directly monitored by
the target, such as DNS server logs, is seldom reviewed or retained. Because the information needed can be
queried using a defined systematic and methodical approach, its collection can be automated.

Employing comprehensive reconnaissance applications
 

Although Kali contains multiple tools to facilitate reconnaissance, many of the tools contain features that
overlap, and importing data from one tool into another is usually a complex manual process. Most testers
select a subset of tools and invoke them with a script.
 
Comprehensive tools focused on reconnaissance were originally command-line tools with a defined set of
functions; one of the most commonly used was Deepmagic Information Gathering DMitry could perform
whois lookups, retrieve netcraft.com information, search for subdomains and email addresses, and perform
TCP scans. Unfortunately, it was not extensible beyond these functions.
 

The following screenshot provides the details on running Dmitry on

 

dmitry -winsepo output.txt example.com

 

Recent advances have created comprehensive framework applications that combine passive and active
reconnaissance; in the following section we will be looking more at

Identifying the external network infrastructure
 

Once the tester's identity is protected, identifying the devices on the internet-accessible portion of the
network is the next critical first step in scanning a network.
 

Attackers and penetration testers use this information to do the following:

 

•   Identify devices that may confuse (load balancers) or eliminate (firewalls and packet inspection devices)
test results
•   Identify devices with known vulnerabilities
•   Identify the requirement for continuing to implement stealthy scans
•   Gain an understanding of the target's focus on secure architecture and on security in general
 

traceroute provides basic information on packet filtering abilities; some other applications on Kali include
the following:
 

Application Description
Uses two DNS and HTTP-based techniques to detect load balancers (shown in the following
lbd
screenshot)
 miranda.py Identifies universal plug-and-play and UPNP devices
nmap Detects devices and determines the operating systems and their version
Web-based search engine that identifies devices connected to the internet, including those
Shodan
with default passwords, known misconfigurations, and vulnerabilities
CENSYS.I Similar to the Shodan search that has already scanned the entire internet, with certificate
O details, technology information, misconfiguration, and known vulnerabilities
 

The following screenshot shows the results obtained on running the lbd script against Facebook; as you can
see, Google uses both DNS-Loadbalancing as well as HTTP-Loadbalancing on its site. From a penetration
tester's perspective, this information could be used to explain why spurious results are obtained, as the load
balancer shifts a particular tool's activity from one server to another:
 

Mapping beyond the firewall

Attackers normally start the network debugging using traceroute utility, which attempts to map all the hosts
on a route to a specific destination host or system. Once the target is reached, as the TTL field will be zero,
the target will discard the datagram and generate an ICMP time exceeded packet back to its originator. A
regular traceroute will be as follows:
As you see from the preceding example, we cannot go beyond a particular IP: which most probably means
that there is a packet filtering device at hop 4. Attackers would dig a little bit deeper in understanding what
is deployed on that IP.

Deploying the default UDP datagram option, it will increase the port number at every time it sends an UDP
datagram. Hence attackers will start pointing a port number to reach the final target destination.

IDS/IPS identification
 

Penetration testers can utilize fragroute and WAFW00F to identify if there are any detection or prevention
mechanisms put in place, such as Intrusion Detection System Prevention system application Firewall
 

Fragroute is a default tool in Kali Linux that does fragmentation. The network packets will allow attackers to
intercept, modify, and also rewrite the egress traffic for a specific target. This tool comes in very handy on a
highly secured remote environment.
 

The following screenshot provides the list of options that is available in the fragroute to determine any
network IDs in place:
 
 

Attackers can also write their own custom configuration to perform fragmentation attacks to delay,
duplicate, drop, fragment, overlap, reorder, source-route, and segment. A sample custom configuration
would look like the following screenshot:
 

Fragroute on target is as simple as running fragroute target.com and if there are any connections happening
to the target.com then the attackers will be able to see the traffic that is being sent to the The following
screenshot shows that the IP segments are fragmented as per the custom configuration file:
 
 

Another tool that attackers utilize during the active reconnaissance is This tool is pre-installed in the latest
version of Kali Linux. It is used to identify and fingerprint the Web Application Firewall products. It also
provides a list of well-known WAFs. It can be listed down by typing the switch -l to the command (for
example, Wafw00f
 

The following screenshot provides the exact WAF running behind the web application:
 
Enumerating hosts

Host enumeration is the process of gaining specific particulars regarding a defined host. It is not enough to
know that a server or wireless access point is present; instead, we need to expand the attack surface by
identifying open ports, the base operating system, services that are running, and supporting applications.

This is highly intrusive and unless care is taken, the active reconnaissance will be detected and logged by the
target organization.

Port, operating system, and service discovery

Kali provides several different tools useful for identifying open ports, operating systems, and installed
services on remote hosts. The majority of these functions can be completed using Although we will focus on
examples using the underlying principles apply to the other tools as well.

Port scanning

Port scanning is the process of connecting to TCP and UDP ports to determine what services and
applications are running on the target device. There are 65,535 ports each for both TCP and UDP on each
system. Some ports are known to be associated with particular services (TCP 20 and 21 are the usual ports
for the file transfer protocol service). The first 1,024 are the well-known ports, and most defined services run
over ports in this range; accepted services and ports are maintained by IANA

Although there are accepted ports for particular services, such as port 80 for web-based traffic, services can
be directed to use any port. This option is frequently used to hide particular services, particularly if the
service is known to be vulnerable to attack. However, if attackers complete a port scan and do not find an
expected service, or find it using an unusual port, they will be prompted to investigate further.

The universal port mapping tool, relies on active stack fingerprinting. Specially crafted packets are sent to
the target system, and the response of the OS to those packets allows nmap to identify the OS. In order for
nmap to work, at least one listening port must be open, and the operating system must be known and
fingerprinted, with a copy of that fingerprint in the local database.

Using nmap for port discovery is very will be detected and logged by network security devices. Some points
to remember are as follows:

• Attackers and penetration testers focused on stealth will test only the ports that impact the kill chain they
are following to their specific target. If they are launching an attack that exploits vulnerabilities in a web
server, they will search for targets with port 80 or port 8080 accessible.

• Most port scanners have default lists of ports that are scanned-ensure that you know what is on that list
and what has been omitted. Consider both TCP and UDP ports.
 • Successful scanning requires a deep knowledge of TCP/IP and related protocols, networking, and how
particular tools work. For example, SCTP is an increasingly common protocol on networks, but it is rarely
tested on corporate networks.

• Port scanning, even when done slowly, can impact a network. Some older network equipment and
equipment from specific vendors will lock when receiving or transmitting a port scan, thus turning a scan
into a denial-of-service attack.

• Tools used to scan a port, particularly are being extended with regard to functionalities. They can also be
used to detect vulnerabilities and exploit simple security holes.

Writing your own port scanner using netcat
 

While attackers utilize the proxying application and Tor network, it is also possible to write their own
custom network port scanner. The following one line command can be utilized during penetration testing to
identify the list of open ports just by using netcat:
 

while read r; do nc -v -z $r 1-65535; done <iplist

 

The same script can be modified for more targeted attacks on a single IP as follows:
 

while read r; do nc -v -z target $r; done < ports

 

The chances of getting alerted in any intrusion detection system using custom port scanners is high.

Large scale scanning
 

In case of large existence of the target, large scale scanning is engaged. For example, with a global company,
often a number of IP blocks exist as part of external internet facing. As mentioned earlier in
Chapter Open Source Intelligence and Passive attackers do not have time-limitations to scan, but penetration
testers do. Pen testers can engage multiple tools to perform the activity; Masscan is one of the tools that would
be engaged to scan large scale IP blocks to quickly analyze the target alive hosts. Masscan is installed in Kali by
default. The biggest advantage of Masscan is randomization of hosts, speed, flexibility, and compatibility. The
following screenshot provides a Class C scanning network within a few seconds to complete and identify the
available ports and services running on the target hosts: