Database Security 3.0

Patch 'em up, Patch 'em up....

By Josh Shaul

Every couple of weeks I hear a complaint from a DBA that they have been flagged for a security violation for not running the latest database patch. It usually goes something like:

“I got written up for a high-risk vulnerability because I’m running one or two patches behind the database vendor. I think it’s unfair and more to the point just plain wrong. I checked out the readme files for the missing patches, and there is no listing of any security fix at all, or at least not one that applies to my system.” 

At first glance, this sounds like a pretty solid argument. Why should someone get written up for a serious security violation for missing a patch when it appears that the patch has no security implications? The problem is, sometimes a patch is more then it appears or claims to be. Database vendors, actually nearly all software vendors release patches for their software periodically. Generally there is some description of the patch, what it’s for, etc…. The part that most folks don’t realize is that software vendors make no promises about fully disclosing what changes go into each patch. Some companies choose to document all changes, however most choose to document only the changes that are directly related to issues reported by customers. Issues uncovered internally are generally patched without any notice. This includes both functional bugs and security vulnerabilities.

There was recently a good example of this with an Oracle July 2008 CPU. In that CPU at least one undocumented issue was fixed, a flaw with Database Vault that allows a DBA to bypass the vault protections and gain access to all the data in the system. If you’re not familiar with database vault, it’s a special kind of Oracle database designed to segregate database management from the data being managed. Basically it’s a tool to keep DBAs from looking at sensitive data in the database. The flaw that Oracle corrected made it trivial for anyone with DBA rights to gain access to any data in the database. 

This is important to know about. The vulnerability Oracle fixed is a fairly big one….but it only impacts data vault. In the grand scheme of things, for the database community at large this was a fairly minor issue. Maybe that’s why Oracle chose not to document it. Not enough deployments of database vault to make it worth their while? Embarrassing to fix a vulnerability in a security bolt-on for the database? Who knows. Thing is, if you’re running database vault to protect your organization’s crown jewels, I’m sure you would want to know about this issue. Otherwise you may choose to skip the patch….and that’s really the point of this post. 

Those complaints that I get about missing patches causing security violations always get the same response: “You can’t trust the vendor to fully disclose what a patch fixes, so you should assume the worst case and patch if you want to ensure your systems are protected.” That’s the bottom line. Missing patches, no matter what the readme docs may say, should always be viewed as a security violation. Not doing so leaves you open to get bit on the you know what……. 

So if Oracle didn’t disclose the vulnerability or the fix, how do we know about it? Well, there is the folly of the partial disclosure approach. Some smart security researcher  (Alex Kornburst) found the vulnerability on his own. He reported it to Oracle, and they responded by telling him that this was a known issue that was (silently) fixed in July 2008 CPU. Oracle is getting better about this kind of stuff, but they still have some room for improvement. I hope that as their security processes continue to grow, that they’ll become much more open about what security issues they are fixing with each patch.

It’s got to be a little embarrassing when one of these “secrets” gets out. Oracle has taken so many big steps to improve their perception about being very serious about security (no doubt Oracle is VERY serious about security)…but when this kind of thing happens, it reminds folks of the dark days when security issues were swept under the rug, and that’s not good for anybody…… 

Interested in learning more? Check out Alex’s blog post about this and another issue with database vault:

http://blog.red-database-security.com/2008/11/21/oracle-database-vault-privilege-escalation-exploit-published/ 

Happy New Year everyone!

 

 

 

And the Survey Says......

By Josh Shaul

This week we released the results of our “Database Security Controls” research study, which was run for us by the analyst firm Enterprise Strategy Group (ESG). We sent ESG off to interview global IT decision makers located in North America. They met with 179 companies and asked a set of questions around data and database security. The responses were collected and aggregated for the report. The results are simply astonishing.

A whopping 84% of respondents believed that they have adequate protection in place for sensitive data.

With so many data breaches reported this year, we were surprised to hear that so many folks felt comfortable with the controls they have in place….but if 84% of companies really have adequate data security, we’re all in pretty good shape….Right?

The survey then drilled down into more specific questions. Have you failed an internal audit? Have you failed a compliance audit? Have you experienced a data breach recently? The answers to the specific questions painted a very different picture. More than 50% of the organizations surveyed experienced a data breach in 2008. I repeat, more than 50% experienced a breach this year, while at the same time, 84% claim to have adequate security. Adequate means enough to get the job done…..Right?

And it’s not just data breaches.

48% failed internal audits. 42% failed payment card industry audits. 38% of federal agencies surveyed failed FISMA audits. The number of failures present a huge problem on their own, but couple that with the false sense of security that the IT community has built around its data, and we really have something to worry about. The amount of data at risk is staggering. For those of you who have experienced a data breach this year, or who regularly fail key components of your audits, stop feeling secure. If you haven’t done something drastic to correct the situation, you’re likely a soft target just waiting to be spotted.

There are signs of hope.

There were some real signs of hope within the survey results. 76% of the organizations we spoke with said that they plan to place purchasing priority on database security solutions in 2009. Wait, didn’t 84% say they already had adequate security?? Maybe folks aren’t burying their heads in the sand after all.

Grab a copy of the survey results from our webpage.

Beanbags and Database Security. A discussion at CSI 08.

By Josh Shaul.

The week before Thanksgiving I led a session at the CSI 08 conference in Washington, DC. It was setup as a group discussion in a small room with bean bags instead of chairs. It was a relaxed atmosphere designed to foster a philosophical discussion. In reality, there was one row of chairs at the back of the room. Everyone except one or two folks sat on chairs. The bean bags were a little intimidating. Once you sit in one of those things, there is no telling if you’re ever getting up. 

Decision Making - The Stakeholders

Our discussion was an exploration of who the stakeholders and responsible parties are in a typical corporation’s database security program. We started by reviewing a few recent data breaches, discussing how they happened and could have been prevented. With an understanding of the mechanics of a data breach, we began on discussion on responsibilities. 

It quickly became clear that not everyone in the audience knew of a database security program at their office. Some even came clean, admitting that they have no such program or effort. Those who did have a plan in place discussed radically different approaches and experiences.  

Group Authority?

A security manager at a large retailer described how his team was responsible for ensuring that databases and other applications are configured securely; in accordance with the requirements of internally developed standards. That was the good news. The bad news was that his group was not responsible for making sure databases had security patches applied. That was someone else’s responsibility. His group was also not responsible for auditing access to databases or monitoring for attacks. Those activities would also fall into other groups….if they were in place at all…he wasn’t sure.

Split up Responsibilities

Another member of the audience described that they have a policy that mandates some basic database security requirements such as strong passwords and access controls. The policy is fairly solid, but there is no one responsible for ensuring that the policy is implemented. Some systems are likely in compliance, others are not. The only people who have a chance of knowing are the system owners, and the scope of their knowledge is very narrow, they only know about the systems they personally own and manage. 

Clear ownership, No ownership

We also heard from a utilities company where there is clear ownership of policy development, and database assessment and auditing, but nobody responsible for remediation or risk reduction. This group made an investment in technology and processes to measure their security posture, but gain very little return from that investment because they rarely fix the problems they find. 

Does anyone own database security? 

According to the group I met with, no one person ultimately does. It boils down to a shared responsibility between operations teams (database admins), IT security teams, and often audit / compliance teams. The group agreed that picking a vocal leader for each of the major functions of a database security program (policy development, assessment, auditing, monitoring, and remediation) is a critical step.

Ensuring that the leaders communicate and cooperate, and are held accountable when their functions don’t deliver is the ultimate key to a successful program, but is often the biggest hurdle to overcome.

Infection Kits and Your Databases: Are You Serving Up Viruses?

By Josh Shaul

During my regular presentations and speaking events on database security, I often discuss emerging threats and trends. It’s a constantly changing landscape, so these new threats are also constantly changing. One that I have been discussing a bit recently are new virus infection distribution mechanisms that rely on a database serving up viruses to unsuspecting clients.

The way this type of attack works is via a web site that stores its data in a database. An attacker exploits a vulnerability in the web site that allows her to get direct access to the database (usually SQL Injection (PDF) or Cross-Site Scripting) and then drops a virus, or more likely a link to a virus into the database. The goal is for the web site to then serve up that link or virus to anyone who visits.

It’s a simple attack that can have devastating results. Nobody expects that when they visit a legitimate website that they could become a target of an attack. When it’s a popular website, the problem can become quite widespread within a very short period of time. It’s a scary scenario, but it is also real and happening today.

We recently saw just this kind of attack occur at BusinessWeek. An attacker exploited a SQL Injection vulnerability on their website and used it to embed java script into the web site’s backend database. That java script would run on any client that connects to the BusinessWeek site and would attempt to download and run malware hosted on a server in Russia. You can read all about it on the net-security website.

It’s a great example of a new and devastatingly effective distribution mechanism for viruses or malware, and it all could have been prevented with more effective security measures. The website could have been designed with security in mind, and therefore proper input validation that could have blocked the SQL Injection (PDF) attempts. A web application firewall could have been in place in front of the website, also blocking the SQL Injection attempt. The database could have been protected with a secure configuration so that the application didn’t have privileges to embed the java script, or so that the SQL Injection attempt could have been logged and kicked off a security alert, bringing in the troops to end the problem before it really got started.

Ideally, all these solutions would be in place. A layered defense is the only way to protect your valuable assets. This incident really demonstrates that the risk of an unprotected database is more then simple data theft. Your database could be serving up viruses and you (like the folks at BusinessWeek) may not even know it.

Regardless of whether it’s your data or your reputation, if you operate on the internet, you’d better protect what you value or someone will come and take it from you. To learn more about database security, visit our website at www.appsecinc.com.

Talking Insider Threat to Databases at ISACA Network Security

By Josh Shaul

This week I gave a presentation about addressing the insider threat to database security at the ISACA Network Security conference in Las Vegas (thanks ISACA for having me!). My session covered database security issues in general, then focused in on the threat from insiders and what organizations can do to protect themselves. It was a very well attended and interactive session. Great to see the audit community showing such interest in data and database security issues!

The real interesting parts of these presentations for me are the Q&A session that happens at the end of my talk. The types of questions that get asked provide great insight into the data security problems that folks are dealing with day in and day out. This time around, the most interesting questions were asked after the session during some one-on-one discussions. There was one question in particular I'd like to discuss:

My DBAs tell me that they have enabled logging on their databases and therefore their DB security work is done. They won't consider adhering to any best practices for configuration, nor are they willing to discuss patching vulnerabilities or fixing weak passwords. What can I do?

This came from an internal auditor at a large public company. Sadly, it's not that unusual a situation. The auditor is rightfully concerned about potential security issues in the databases that could lead to major data loss. The DBAs however want nothing to do with it, claiming logging is more then enough. They couldn't be more wrong.

Logging is important. It provides an audit trail of what actions have been taken in a database. Logging however does nothing to improve the security of the database. If weak and default passwords are in place, logging won't stop someone from breaking in and stealing data. If vulnerabilities exist and can be exploited, logging won't stop an attacker, in fact depending on the vulnerabilities being exploited, logging may not even notice what's going on. Furthermore, logs are typically under the control of anyone with DBA privileges. This means both the legit DBAs and anyone who has gained unauthorized DBA rights. Any of these folks can disable the logging system or erase the logs.

Information security is all about layers of defense, and the first layer in a database must be a secure configuration. Eliminating the weak passwords and establishing password controls, enabling security features, applying security patches, and practicing the principal of least privilege; these are the elements that bring real security to the front lines of a database system. I encouraged this auditor to escalate the issues, and to come up with a set of security configuration standards based on accepted industry best-practices for securing databases for her DBAs to adhere to.

We also discussed how logging is different from Database Activity Monitoring, and that logs are only useful if they are frequently and carefully reviewed by someone who can differentiate between normal access and suspicious or malicious access. A weekly review of the logs from all systems by one manager is not nearly enough. Even if an attack was captured, a week later it is too late to do anything about it...if indeed that attack can be identified by a quick examination of the logs.

Database Activity Monitoring would deliver much more value to their organization. By identifying and alerting on suspicious or malicious behavior in real time, and feeding that info into a centralized security event monitoring system, they can take action when something bad happens. Even with a DAM solution in place, the need for a secure configuration and rigorous standards do not go away. Again we go back to the layered defense, and start with the basics. It’s just like making sure to lock the door and close the windows before you enable your alarm system.

Let common sense rule when securing your data! Lock down your systems first, then monitor to make sure the controls you have in place are effective.

If you have questions about securing your databases, ask us at: asktheexpert@appsecinc.com

It's the Privileged Activity You're After....Not Just the Privileged Users

By Josh Shaul

It’s a problem that I see organizations struggling with every day. An audit finding has come down requiring the monitoring of privileged users in databases containing sensitive or regulated data. From the outside, this seems simple. Get a list of privileged users and log whatever they are doing. However, as folks dig into this, the first question they have to ask is who are the privileged users? And that’s where the trouble starts.

What exactly is a privileged database user? It’s probably not anyone and everyone with access to the system. Maybe it’s only the database administrators, but what exactly does that mean? Membership in a DBA or SysAdmin role? That can be very misleading, since DBA level privileges can be granted directly to a user without any role membership, or new roles with names like Not-A-DBA can be created with DBA equivalent privileges.

Maybe a privileged user is anyone with write access to the database? But that tends to include everyone, since it’s been pointed out by database security researchers many times in the past that creating a read-only database user is essentially impossible, due to all the write privileges that are typically (and often irrevocably) granted to PUBLIC (that’s everyone in the database).

Even with a firm definition of what makes a user a privileged user, the problems don’t go away. Just because a user is not privileged today, does not mean they won’t be privileged tomorrow. Actually creating and maintaining an accurate list of privileged users in an enterprise full of databases can be an impossible task.

There is another way forward though. One that is simple to define, implement, and maintain. One that meets the letter of regulations and audit findings, and one that ensures that no matter how or when a user obtained their privileges, that their activities in the database are tracked. The answer is to pay attention to the activity first, and who did it second.

While it’s hard to define what a privileged user is, it is easy to define what privileged activity is. Changes to the structure of the database (DDL) and changes to the privileges in the database (DCL) are both clear areas of privileged access that must be tracked. Changes to data (DML), particularly changes to sensitive or regulated data is another clear area of privileged access. Perhaps direct connections to a database via an ad-hoc query tool, or via MS Access or MS Excel are considered privileged access. Check out the latest Insider Threat White paper in the Application Security resource center.

In the end the exact definition of what is and is not considered privileged access is going to be dependent on the organization, the auditors involved, and the regulations that govern protection of the stored data. However, in every case, these definitions are fairly straightforward to establish, and once they are have been laid out, it’s easy to implement monitoring based on the activity type, instead of based on the user running queries.

There are a number of ways to actually implement this monitoring. All DBMS vendors provide some native auditing functionality. For the major database vendors, these audit systems allow configuration based on the commands being issued, not just the user issuing them. Native database auditing systems have a poor reputation though as a performance nightmare, so DBAs often fight hard against enabling them. There is another equally important reason not to rely on native database auditing. Those systems are under the complete control of the DBA, the exact people that we’re being asked to monitor. It is hard to prove the integrity of an audit trail when the persons being audited can enable, disable, and reconfigure the audit system at will.

A much better approach is to use a 3rd party database activity monitoring (DAM) tool. These products tend to have little to no impact on database performance, and the good ones on the market are setup such that they are not under the control of the administrators being monitored. I fully expect that in the not too distant future, IT auditors will require the use of trusted 3rd party systems to implement privileged activity monitoring, purely for the segregation of duties benefits that they bring.

If you’re facing a situation where you’ve been required to monitor your privileged users, make things simple for yourself by turning the problem on its head and monitoring the privileged access.

For more information on Database Activity Monitoring, visit the Application Security, Inc. website at www.appsecinc.com.

If you are serious about monitoring priviledged users and activity, I definately recommend checking out the Arrest the Threat white paper.

Talking Patching at Microsoft TechEd

By Josh Shaul

Last week I hosted a Birds of a Feather discussion at Microsoft TechEd in Orlando. My session was titled “A Practical Discussion about Protecting your Enterprise Databases” and focused mainly around MS SQL Server and Oracle (running on Windows of course). This was a pure group discussion, where I was only acting as a moderator and facilitator. Nearly all of the 40 something people in the audience were Database Administrators (DBAs) who have been tasked with securing the databases they manage. 

The first topic in our 75-minute discussion was framed around database patching. It was enlightening to hear about the different strategies that have been employed by the various organizations represented in the room. The most common approach was to wait for the vendor of the application that uses the database to certify that their application works with a patch before that patch gets applied to a database. The point was quickly made that application vendors typically don’t certify their apps at every patch level, rather they wait for larger service packs (in Microsoft terms) or Patch Sets (in Oracle terms). These larger patches usually come out no more then once or twice in a year. A DBA at a software application vendor at the meeting spoke up to confirm that his employer does just this, only putting a product through QA after a major patch release from a database vendor.

With the litany of database vulnerabilities that are patched every quarter, this leaves an organization’s systems exposed to a variety of dangerous exploits. Recently, it has become easier and easier to find exploit code for recently patched vulnerabilities. Often it takes no more then 24 hours after a database vendor has released a security patch to find exploits for most if not all of the issues that have been corrected. If you patch once a year, you spend at least 75% of your time exposed to what are often devastating vulnerabilities.

Another approach that several folks took was what I like to call patch and pray. Relying on their skills at rolling back patches, these folks take and apply the database vendors patch releases as quickly as they can get them installed. There is no testing or pre-qualification work, they simply accept the patches, install them on their database servers, and pray that everything continues to work…..and often things do continue to work. When there are failures, things get ugly. It’s emergency patch roll-back time. Of the folks in the audience that use this approach to patching, it seemed like everyone had at one time or another been called in for an emergency patch removal. This method provides quick coverage for newly fixed vulnerabilities, however it is simply not appropriate for systems with very high reliability or uptime requirements. 

The last popular approach was avoiding patching entirely, at least until a system has been affected by not having the patch. If a system hits a bug and fails and there is a patch available to fix that problem, then and only then is patch application considered. In this approach security patching is never a consideration, unless a vulnerability has been exploited, damage done, and the attack traced back to the missing patch. This reactive approach leaves systems vulnerable to attacks that may be quite old and well known, leading to low hanging fruit for an attacker to go harvest.

There were one or two folks who discussed a more methodical approach to accepting, testing and applying patches to production. These folks apply patches quickly to their development environments and start running some tests to see how things go. If there are no problems, the patch is applied to QA and tested more rigorously. When all tests pass, the patch is cleared for installation in production and a change window is scheduled to apply the patch. While this approach can leave a significant gap in time between patch release and application to production (the Patch Gap), it is a proactive approach to patching that brings system level security with low risk. In my view this is the best approach, particularly if combined with a Database Activity Monitoring system that can monitor during the patch gap and provide alerting in case an attempt is made to exploit a known vulnerability.

Despite the different approaches, different requirements and widely varying industries represented in the discussion, there was quite a bit of commonality that we all found comforting. Every organization has the same struggles with patching. It’s not easy, it’s not quick, and it rarely comes without some pain. But there was a consensus among the group that security patching is a good cause, and as DBAs they want to be good stewards of the data in their systems. That’s a good sign. As administrators take personal responsibility for the security of their systems, everyone benefits…..everyone except the criminals that is……

InfoSecurity Europe – Security Biz Abuzz Across The Pond

Submitted by Ted Julian

This week I’m part of an elite AppSec US / Europe joint task force manning the InfoSecurity Europe show in London. If you’re familiar with the RSA conference in the U.S., this show is essentially the European equivalent from a tradeshow perspective (there are far fewer speaker sessions).   Given that RSA was just a few weeks ago one can’t help compare the two. 

Here are some impressions:

• McAfee two booths, Symantec none.  My theory is that one of McAfee’s acquisitions bought a booth some time ago and McAfee decided to keep both (or couldn’t back out without paying a ton).  In any case, the way they used them was brilliant.  One booth handed out the left “compliance sox” (a black sock with a snazzy McAfee logo on it)
  the other the right.  For a complete set you had to visit both booths.  Pretty clever approach for two booths and one that certainly sent a message about their commitment to the European market.
• Sarbanes matters in Europe, too.  McAfee’s “sox” gambit deserves extra credit for a credible tie-in between their product set and a critical business issue, Sarbanes compliance. Yup, Europe cares about Sarbanes, too – it’s been a major issue at every customer meeting so far on this trip.
• Database security, representation.  Counting us, there were at least three database security vendors with booths and another that was there to present.
• Speeds and feeds are in.  I was shocked to see that many booths were very much focused on their product feature set and value proposition as opposed to higher-level business drivers (ex. lower transaction costs through PCI compliance). And judging by booth traffic, attendees liked it that way. In contrast, booths at RSA (as others have pointed out) are so high-level they risk losing touch with reality.

As with RSA, there aren’t any obvious themes at InfoSecurity Europe. But interest is high, the show floor has been consistently packed.