Update: Please note this article was first published in July 2012. While we stand fully behind this post as true & accurate at the time of publication, we make no representation as to current applicability.
As the head of the engineering group at a leading provider of healthcare information technology, I frequently encounter the need to transfer files containing protected health information both internally amongst colleagues and externally with clients. Exchanging files containing data subject to HIPAA in a compliant manner is challenging with homegrown solutions and as Chief Security Officer I am directly responsible for meeting that challenge. So almost nine months ago we began evaluating market solutions, and liked what we saw in Egnyte.
The fast-growing “hybrid” cloud company, which just closed a $16 million investment from Google Ventures, Kleiner Perkins Caufield & Byers and Polaris, markets “HIPAA compliance for healthcare, pharmaceutical and biomedical businesses” and offers to execute a Business Associate Agreement during account creation. With such a strong pitch for HIPAA compliance and the association with big Valley-based venture capital, one might be forgiven for trusting Egnyte to securely store and manage highly sensitive structured data without any additional due diligence.
However if you were inclined to probe further, you would find, as we did, that their posted policies seem adequate, that they appropriately disable SSLv2, and that their security whitepaper makes bold claims like “Egnyte prevents cross-site request forgery and cross-site scripting” and _“Egnyte employs a third-party security firm to perform continual penetration tests to confirm the stability and reliability of the system”. _You might even note how the whitepaper states that user passwords are protected with bcrypt – a password encryption mechanism that should appease even the most paranoid among us.
Satisfied with the findings of the most thorough evaluation we could conduct externally, I directed a company-wide rollout of Egnyte to serve as our HIPAA-compliant cloud storage and transport service. While the service met our immediate needs I desired a tighter integration with our SaaS platform, and Egnyte’s rudimentary API proved inadequate for any sophisticated integration effort.
To meet that need, my co-founders and I decided to build an alternative IaaS solution complete with a fully open API. Exporting our files and folders from Egnyte and into this new service, which we named FolderGrid, was simple enough since Egnyte exposes an FTP interface. But Egnyte makes it difficult to export your users and permissions – and our effort to migrate our users and permissions is where this story gets interesting.
My team used Google’s Chrome browser to examine the Users & Groups management pages on Egnyte’s service – hoping to find an easily parsable form of our user and group data. Here’s a screenshot of where we started (with identifiable data blurred to protect the innocent):
Like many modern web applications, Egnyte’s web interface uses XHR to pull data from the server. So we opened Chrome’s Developer Tools window to examine the XHR URLs used to populate this screen:
The first request’s named certainly seemed like it might hold the data we were looking for so we started there by examining the server’s response:
We stared at that response in disbelief – much like I imagine you are doing as you read this. For the uninitiated, MD5 hashes have been widely regarded as dangerously insufficient for years. So the idea that Egnyte might still be using MD5 hashes was troubling, but to see any password hashes (MD5 or otherwise) appear exposed to endusers is a critically serious security defect.
This request was initiated as we were viewing our Power Users so we jumped over to the Standard Users screen and saw more of the same:
I immediately logged out of my administrator account and logged back in with a normal “Power User” account with no administrative privileges. I was able to reproduce the calls and pull a full listing for every user in the account – including the administrator accounts.
So although the screen rendered by my browser only displayed basic information such as user names, email addresses, and role – the server was actually delivering far more data about every user and that data was readily readable by any Power User on the account. The obvious next move was to determine just what was included in that broader set of user data.
Upon examination there were 100 fields for every user. This screenshot only shows some of the more interesting fields we discovered:
On a hunch, I pulled out all the MD5Passwords from my user listing and decided to run a hash search on each to determine if any were easily crackable:
The very first password hash in my list scored a hit using a specialized Google search:
Although that hit alone was likely evidence enough to prove Egnyte was using unsalted MD5 hashes for password encryption, I decided to run one more test. I changed a user’s password and set it equal to my own. Then I reloaded the user screen and examined the new XHR response – our MD5Password hashes were identical.
If you’re having trouble following the technical details – the implication of all this is that any Power User on my Egnyte account could, with relative ease, deduce any other Power User’s username and password – including Power Users with administrative permissions. This of course defeats any and all permission controls on the account as well as any audit trails – since the malicious user would be indistinguishable from the authorized user once authenticated.
If you are an Egnyte account administrator you should immediately disable the option allowing Power Users to interact with Users & Groups (and of course change all of your Power Users passwords). That configuration change should protect you from this privilege escalation vulnerability. It was the first action I took – right before I began removing all data from Egnyte’s service.
As the Chief Security Officer of Pascal Metrics, I’m ultimately responsible for the safe-keeping of highly sensitive patient data from the biggest & most trusted brands in healthcare. Deciding to entrust Egnyte with some of this data was not a call I made lightly and I’m disappointed to find such a negligent oversight in the design of their web interface. Fortunately, we discovered these material weaknesses in Egnyte’s service and took corrective action before putting our clients at significant risk.
Rapidly evolving software will never be defect free and perhaps Egnyte could be forgiven for unintentionally exposing its customers to this privilege escalation vulnerability. After all, the configuration workaround I described offers some immediate protection. However, by adopting a cloud service an end-user cedes enormous amounts of control to the provider of that service – an act requiring a high degree of trust. By their nature, cloud services are not easily externally vetted and customers must rely upon the stated policies and claims of a vendor to evaluate the vendor’s offerings.
The vulnerability I have described allowed us unusual insight into the detailed workings of Egnyte’s service. They are clearly using unsalted MD5 hashes for password encryption (an indefensible design) while claiming to adhere to much higher standards (bcrypt). This blatant misrepresentation undermines any trust I might have retained for the service.
We’ve stopped trusting Egnyte, and we’ve stopped using their service. Given the potential ramifications of a data breach – especially one involving data subject to federal regulations, my advice to fellow CSOs (and all users of cloud services) is: caveat emptor.