Enterprise HTTP Security Inspection For Penetration Testing

The need for HTTP Security Inspection on Application Security

Application Layer – HTTP from the Security Perspective

An Application Layer is the first layer which need a security check which just goes beyond any other common checks. Somehow, automated scanners might do this as pre-defined in the programmed logic, but most of them fail to find the bugs which passes through the HTTP Handler and hence create critical vulnerabilities for business enterprise.

Yes, I had been talking about Hyper Text Transfer Layer Protocol, which is by now the most common, wildly wide used text based protocol around the internet. Web Applications use the text based protocol since it’s easier to implement and handle parallel requests. The Web Server handles these requests which are made by the clients and penetration testers often ignore a variety of checks against HTTP from the security perspective.

Once, an application penetration is fairly grown studying HTTP at a deep level, he/she could already understand why a particular request could be manipulated by client side proxy in multiple of ways and produce a critical security bypass.

Before I jump into concluding the security aspects of the Hyper Text Transfer Protocol, my application security research have shown a comprehensive study of where to start from HTTP from the security standpoint. This question has been asked a lot of times and people have failed to come up with an exact method to detail everything in one book or post.

This study is more of a guideline for application penetration testers rather than a reference study; but either way it could be used both the ways. I had prepared a standard draft for HTTP Security Research in topic segments which if wished could be sub-categorized but for the sake of the reference and a guide, i have detailed them into modular topics which could be used by any application security researcher, bug hunter, or application security enthusiast for their own analysis.

To detail each and every aspect of the ongoing research at Defencely, I have made the results available and the technical study of the topics which could be found extremely helpful for new application security beginners as well as experts since some vulnerabilities or critical web application related security access control bypasses rely on the HTTP protocol study.

Certain aspects of HTTP handling and application related vulnerabilities are around inspection of HTTP Protocol and is easy since HTTP Protocol is implemented to be text based and hence easily manipulated by penetration testers.

Why HTTP data and headers are manipulated?

The HTTP Protocol was built by IETF (Internet Engineering Task Force) to communicate with different nodes using hyperlinks which could be distributive and at the same time flexibly collaborative. Since the protocol was developed at the very early stages of the internet and there was a requirement for a lightweight protocol to supported distributive and collaborative nature altogether, this gave rise to ‘text’ based communication with certain set-rules (protocol) which both the nodes have to follow.

HTTP data or its headers could be attacked or manipulated for many reasons:

1. It’s plain text which could be easily manipulated by a client side proxy.
2. It’s widely used and is an extensible protocol and hence browsers adapt to it.
3. Its clear text until it is HTTPS or TLS (Transport Layer Security) added to it.
4. It passes certain security tokens to remediate attacks such as XSS and XSRF.
5. It’s plain text and hence could be easily intercepted by a proxy reading data.
6. It sends POST body which is often manipulated in file upload attacks.

Apart from the above, there could as well be many other security considerations which have to be taken from the application security perspective.

Along with HTTP being plain text which is why it is manipulated by client side proxies, it’s also possible an attacker could just sit in-between the two nodes and intercept all the sent request to later replay the packets and gain authorized access to an application if security tokens which should be validated server side were not being validated at all.

This way, it is fairly possible to bypass any client side restrictions since the penetration tester or an attacker has full control of it. If security tokens which were passed along with HTTP headers were validated server side, this restriction could also be possibly bypassed with HTTP security in mind. The latter is the reason why HTTP deep inspection is needed for any budding application security researcher or an application security penetration tester.

HTTP Security Headers

To begin with HTTP Headers from the application security perspective, one must go through the following HTTP Headers which has a considerable impact on overall application security:

• Strict-Transport-Layer
• Frame-Options
• Content-Security-Policy
• Content-Security-Policy-Report-Only
• X-Content-Security-Policy
• X-Webkit-CSP
• X-Content-Type-Options
• X-XSS-Protection
• X-Frame-Options

The above HTTP Security Headers are implemented to mitigate attacks such as:

1. Clickjacking/UI-Redressing
2. Cross Site Scripting (XSS)
3. MIME-Sniffing

These above attack patterns and their sub-variants are strictly bound against HTTP Header Security. Anything else as advanced as Cross Origin WebSockets Attacks such as Cross-Site WebSocket Hijacking, and other similar attacks are entirely related to different protocol (Web-Service protocols) which API’s have been using now-a-days. From an application security perspective, there is a string need to change the general application audit methodology to test HTTP Security as part of the manual test which could be then resulted in a variety of reporting standards.

One such reporting section on the engagement/penetration test report could be ‘Protocol Layer Vulnerabilities’ which do not really related to be purely application code level vulnerabilities but rather assisting several other vulnerabilities resulting in a critical security impact on the application covered. Some of the most abused HTTP Security have been mentioned in the sections below for your ready reference.

The Business Logic Vulnerabilities which do not rely on code level bugs could be related to be HTTP weaknesses and hence could assist in providing an additional surface layer for application logic exploitation based on the scenario. The attacker’s perspective is required to define the logical vulnerabilities.

HTTP Information Leakage

HTTP being an extensible layer and having been implemented as a clear text protocol which have methods, body and commands in ‘plain text’ could be most intercepted and then manipulated via client side proxy, or any other proxy such as ISP in-between who could easily read personal information, etc.

A large amount of data such as usernames, passwords, encryption keys, location, etc. are used by applications for details regarding the user. This alone is the reason, all of these critical traffic needs to be encrypted along the channel somehow. This is done by using a TLS (Transport Layer Security) which could mitigate the issue if critical enterprise business data have to be transferred which could be employee information, contract details sent from a POST Request (in this case, the REQUEST Body have to be provided with some level of security).

Here are some key-points which should be considered when related to HTTP Information Leakage as a security concern:

• When confidential information is transferred via one source to another, the destination source (which could be a server) should store the retrieved information in an encrypted form.
• At some point, server side software reveals their version (also known as banner grabbing from the penetration testing perspective). This might allow the server machine to become more attractive to botnets and hence attract attention of the malicious attacker controlling the botnet.
• Proxies which serve as a portal through a network firewall should take special precautions regarding the transfer of header information that identifies the hosts behind the firewall. Encoding is done here. But that’s not what exactly security is all about. Encoding can never be compared to Encryption, it’s the first hand rule.
• The information sent in the form field could be a conflict with the users/customers privacy interests and hence this information should be sent in a more secure way or there should be scope of allowance of what information the user ‘wants’ particularly to submit. If a particular information is anyway essential for the application to work with and produce results, this information essentially should be transferred securely.
• Using the Referer header sent by a client in a non-secure HTTP Request to a page with a secure protocol set.
  Submission of any sensitive data should not be triggered using GET method based form, the data sent could be clearly intercepted and visible to 3rdparty and is an insecure way. This data however could be converted to its encoded form, in which case, it’s not secure yet. POST submissions are generally a formal approach and is neither a secure way if at all TLS has not been implemented already. Idempotent Methods and non-Idempotent methods are a way HTTP have already defined in its security suites which should be used wisely.

HTTP File Path Traversal Attacks

It’s rare but on my application vulnerability research and penetration test with most generic application as well as enterprise business applications, most of these applications were critically involved into providing attacker an opportunity to guess and dig information using File Path Traversal attacks.

For an example, in UNIX/LINUX or Microsoft based Operating Systems, the characters ‘..’ mean to indicate a directory level above the current directory.

On such a system (wherein the application was deployed with a web-server configured to handle HTTP Requests), the HTTP Handler must disallow such ‘construct’ in the request URI, since this way the traversal attack would work and an attacker could traverse through directories, and hence manipulate path leading into sensitive file disclosures.

The sensitive resources should be restricted to the resources returned by HTTP Requests to be only those they were intended to be served by the web-administrators and not in accordance to the attacker. This have been a very naïve overall description.

There are particular ways to bypass these restrictions and the HTTP handler have to be hardened enough to handle these bypass constructs. The requests coming in from the client side could never be trusted since there is no follow-up yet again after the request has been handled and the attacker had already moved one step forward with what he needed.

HTTP DNS Spoofing

Clients fall prey to mis-association of IP addresses and DNS names since the clients rely on using HTTP and use Domain Name Services (DNS). This way the client has to be conscious about its associativity with Domain Name Service and the validity of the Internet Protocol (IP).

Therefore, as an end-result, the client has to check continuity of using a valid IP address and check its integrity. HTTP clients not following the sent TTL records by the DNS could be result in DNS spoofing attacks when previously-accessed server IP address has been changed.

HTTP Location Headers and HTTP Header Injection

Resources are critically stored in a server which have confidential enterprise information stored. These content however secure it might look, could be accessed in an unauthorized way by spoofing Content-Location headers. Consider a scenario wherein a master server serves different organizations and all of these organization do not trust each other on sharing business related confidential data.

In order for the server to recognize each of the request which have been generated from an organizational client, the request could contain a ‘Location’ header to justify its identity, but this security counter-measure which was original planted to avid security conflicts turns up beneficial for an attacker where he/she could just spoof the addresses and hence authorize a particular resource which was not supposed to be accessed originally and intended by the web-administrator.

A strict access control in this scenario could just be essential for an enterprise business to rely on the security posture. I have went through multitude of test cases, where it is possible to access the most hardest to crack application ‘thinking the different way’, an out-of-box penetration test red team could accomplish this; let the experts take the control and the only way to pin-point these weaknesses is signing up for expertise application security consultants.

Apart from everything else, HTTP Security is a vast topic in itself which needs to be kept in check while testing applications from a security standpoint. From the developer’s perspective, it would be consuming them production time to look at each and every security aspects of an application.

This is the exact reason a specialization service is needed to mitigate potential vulnerable application in safe development environment prior to deploying them without any security impact consideration, especially when an application servers critical service, processes customer information and has a complex web-architectural setup.

Apart from application security, HTTP Security in contrast parallel to the same is a must for any real state-of-the-art Red Team Engagement.

HTTP Research

HTTP should be studied thoroughly by an application penetration tester who could make a day for their living. A lot of critical bugs are in and around the HTTP header manipulation and looking how HTTP requests were handled by a remote we-server. Adding to this, I have compiled some topics which could prove essential to an application security tester in general both from the developers perspective and the application penetration tester perspective.

To study the basic HTTP Protocol, one could start as described here or have a brief look at the below mentioned topics:

• HTTP/1.1 Authentication.
• HTTP/1.1 Caching.
• HTTP/1.1 Range Requests.
• HTTP/1.1 Conditional Requests.
• HTTP/1.1 Semantics and content.
• HTTP/1.1 Message Syntax and Routing.

Add these as well ..

1. HTTP Session Management.
2. HTTP State Management.

Below mentioned topics talk very brief into status codes and ‘codes’ in general with HTTP:

1. HTTP Status Codes.
2. HTTP Redirect codes for RESTful services.

These header study from the very basic approach regarding standard HTTP headers and some Framework added HTTP headers along with their security header and their respective fields could prove essential:

1. HTTP Accept-POST HTTP header.
2. HTTP Header field X-Frame-OPTIONS.
3. HTTP Header filed registrations (this has all the header overview)

All of the below mentioned HTTP topics are related to client side controls, including HTTP signatures.

1. HTTP Signatures.
2. HTTP Client Hints.
3. HTTP Cache-Control Extensions for Stale Content.

Some of the required assets of encoding schemes which will be needed to further study HTTP are given below:

• Character Set and Language encoding for HTTP.

and these for HTTP/2.0 with WebSockets Concepts used in common web-services by RIA (Rich Internet Applications) as well as light API applications:

1. HTTP/2.0
2. WebSocket over HTTP/2.o
3. Minimal Unauthenticated Encryption (MUE) for HTTP/2.0

If you want to go over the additional protocols which have a similarity between them and the HTTP Protocol, refer these:

HTTP Strict Transport Security (HSTS) – Helps in mitigation research studies.
OAuth 2.0 Message Authentication Code (MAC) Token.

Overall, all of these basic topics should be previous studied thoroughly before being conclusive on a given penetration test. All of these HTTP topics could play a very important role in the testers knowledge and knowing how the workflows of an application might just work (given the test is a blackbox), and hence could be guessed for application based attacks via the HTTP Protocol.

About the Author -
The article is submitted by Shritam Bhowmick. He is an Information Security Analyst cum Penetration Tester. He does Application Security. You can connect him via LinkedIn