Operating system fingerprinting is the process of learning what operating system is running on a particular device.
By analyzing certain protocol flags, options, and data in the packets a device sends onto the network, we can make relatively accurate guesses about the OS that sent those packets.

By pinpointing the exact OS of a host, an attacker can launch a precise attack against a target machine. In a world of buffer overflows, knowing the exact flavour and architecture of an OS could be all the opportunity an attacker needs.
Why OS Fingerprinting?
Learning remote OS versions can be an extremely valuable network reconnaissance tool, since many security holes are dependent on OS version. As long as this information is not revealed, the attacker is limited in the variety of attacks and exploits. Therefore the focus on initial information gathering is finding out the operating system.
For example, without OS fingerprinting, an attacker is unable to know whether the target has an IIS server or Apache server, and it’s pointless to try IIS exploits on an Apache server.
TCP and ICMP Fingerprinting
  • TCP Fingerprinting
    OS scanning works with the TCP/IP suite of protocols. TCP/IP is the protocol that the Internet relies heavily on. All the communication on the Internet is done using this protocol suite. This makes it very necessary for an operating system to implement it, to communicate with other machines flawlessly. IP is used to assign a logical address to machines on the network, and TCP is used to transfer IP packets in an acknowledged fashion. These flags are essential for OS fingerprinting, since each operating system reacts differently to normal and specially crafted TCP packets sent to its network stack.
TTL (Time to live) is a value set by the computer or the device that sends an IP packet. Every router that comes in between the packet and its destination will reduce the value by 1. So if a packet has travelled too long, overcoming too many hops (router machine’s in between), and the TTL value becomes zero (because it got reduced by 1 at all hops in between), then that packet is discarded.
  • ICMP Fingerprinting
    Another protocol often used in fingerprinting is the Internet Control Message Protocol. Most traceroute utilities use ICMP to discover the network path a packet takes to its destination. ICMP also returns error messages when a datagram is not processed correctly, whether due to the device not being active on the network or a problem with the datagram itself. These error messages can also be useful.
The header looks different for each request and reply packet though.
Types of OS Fingerprinting
  • Active fingerprinting
Active fingerprinting is the process of transmitting packets to a remote host and analysing corresponding replies. This allows the scanner to obtain more accurate results than a passive scanner, and in a shorter amount of time. The traditional approach is to examine the TCP/IP stack behaviour of a targeted network element when probed with several legitimate packets.
  • Nmap
The first step of network recon is to determine what machines are active on the network. One popular tool that employs such OS detection methods is Nmap, which not only allows you to detect the operating system running on a remote system, but also perform various types of port-scans.
Nmap OS fingerprinting works on the concept of sending multiple UDP and TCP packets to the target hosts, and then analyzing the reply. During the OS scan using Nmap tool, the tool will send requests to both open and closed ports to analyze the reply method. This option tells Nmap not to do a port scan after host discovery, and only print out the available hosts that responded to the scan. This is often known as a “ping scan”. It allows light reconnaissance of a target network without attracting much attention. Knowing how many hosts are up is more valuable to attackers than the list provided by a list scan of every single IP and host name.
Attempting to detect an operating system with Nmap is as simple as running it with the -O switch. Here is the result of scanning a Windows machine.
Below is the result of scanning a Linux machine.
  • Xprobe2
You can also detect the remote operating system by using xprobe2. The purpose of the tool is to perform fingerprinting of remote TCP/IP stacks based on Ofir Arkin’s ICMP fingerprinting research. Xprobe2 is an active operating system fingerprinting tool with a different approach to operating system fingerprinting. Xprobe2 relies on fuzzy signature matching, probabilistic guesses and multiple simultaneous matches, and a signature database.
TCP scans are pretty easy to detect for IDS systems. Hence using only the ICMP modules available in xprobe2 is pretty nice for remaining stealth.
Currently, xprobe2 has the following modules:
  • icmp_ping: ICMP echo discovery module
  • tcp_ping: TCP-based ping discovery module
  • udp_ping: UDP-based ping discovery module
  • ttl_calc: TCP and UDP based TTL distance calculation
  • portscan: TCP and UDP PortScanner
  • icmp_echo: ICMP echo request fingerprinting module
  • icmp_tstamp: ICMP timestamp request fingerprinting module
  • icmp_amask: ICMP address mask request fingerprinting module
  • icmp_port_unreach: ICMP port unreachable fingerprinting module
  • tcp_hshake: TCP Handshake fingerprinting module
  • tcp_rst: TCP RST fingerprinting module
  • smb: SMB fingerprinting module
  • snmp: SNMPv2c fingerprinting module
For fingerprinting a remote machine, you can just call xprobe2 and give the remote machine IP address or hostname.
Passive fingerprinting is the process of analysing packets from a host on a network. In this case, the fingerprinter acts as a sniffer and doesn’t put any traffic on a network. It is called passive because it doesn’t involve communicating with the host being examined. Based on the sniffer traces of these packets, you can determine the operating system of the remote host. Passive scanners are generally and inherently less accurate than active scanners, due to the fact they have less control over the data they are analyzing.
  • NetworkMiner
NetworkMiner is a network forensic analysis tool.
NetworkMiner can be used as a passive network sniffer and packet capturing tool in order to detect operating systems, sessions, hostnames, open ports, etc, without putting any traffic on the network.
For OS fingerprinting we have to run NetworkMiner and select the network interface for which the data has to be captured. We can sort hosts by IP address, MAC address, hostname, Operating System etc. And then click start.
NetworkMiner displays the identified OS for each host in the “Hosts” tab by showing an icon for the OS next to the host in the tree-view. Each host can also be expanded, which enables the user to see a more detailed analysis of the matching OS fingerprints for that particular host.
OS Detection with Ping Command
OS detection could be performed simply by using a ping and determining the OS of the destination host based off the TTL value returned with the ping response.
Above are the values for the more popular operating systems.
Result of scanning Linux machine:
From the below shown example of a ping, you can easily see the TTL value is 128. That’s the default TTL value for Windows. 
Prevention
It is nearly impossible to block all fingerprinting attacks, but we can make it difficult by using several measures. We have to make sure that external hosts are not able to directly scan internal targets.
Active OS fingerprinting can also be solved with use of firewalls and
Intrusion Prevention System.
Banner grabbing should be a bit easier to defend against. The Apache config file allows you to limit the information listed in the header.
If we have some service running and there’s an open port, mask or delete the server information when an error triggered.
Conclusion
OS Fingerprinting is a very valuable technique to find out the OS. From an attacker’s point of view, OS fingerprinting is helpful in figuring out what vulnerabilities the system might have or which exploits may work on a system. There are lot of techniques to do OS fingerprinting, at the same time there are a number of ways to overcome and avoid OS fingerprinting.
References 
  • http://en.wikipedia.org/wiki/OS_fingerprinting
  • http://oreilly.com/pub/h/1347
  • https://www.sans.org/reading-room/whitepapers/testing/overview-remote-operating-system-fingerprinting-1231
http://resources.infosecinstitute.com/overview-os-fingerprinting/