Is Healthcare.gov security now fixed?

Cross-posted from Is There A Problem Here?

I first attempted to use HealthCare.gov to learn about options for covering my granddaughter, who is not covered by my employer-subsidized insurance. I encountered the same kinds of account creation issues others have reported, but I decided to turn on my web browser’s built-in developer tools to see if I might see details as to why form submissions were failing. I quickly discovered that the main browser window would often display a status other than what was actually occurring. For example, the form submission would fail to get a response from the server but the user interface would report that the form was submitted. Once I saw this behavioral mismatch between what was displayed in the browser and what was actually happening, I kept developer tools on as I used the site.

I do not consider using developer tools to watch data moving in and out of my own computer to be “hacking.” I have NOT “hacked” Healthcare.gov. I have only observed what is sent to my computer. I have NOT attempted to gain unauthorized access to Healthcare.gov accounts. Attempting to gain unauthorized access would be both unethical and illegal. Please don't try it.

While watching the interactions between my web browser and the Healthcare.gov servers, I saw information being sent to my computer that likely should not have been sent by the server. After I was told that Healthcare.gov will not take reports of security concerns, I started blogging them.

Then I came across a very serious issue.

I discovered a design defect that subsequently led to me receiving a great deal of media attention. Little did I know that my findings would be mentioned in Wednesday's congressional hearings:

ESHOO: On the issue of security, there was a security breach that arose recently, that I read about at any rate. And what I think is very important here, because the issue of privacy has been raised, and I think that that has been answered. Very importantly, there isn't any health information in these systems. But there is financial information, so my question to you is, has a security wall been built, and are you confident that it is there and that it will actually secure the financial information that applicants have to disclose?

SEBELIUS: Yes, ma'am, I -- I would tell you that there was not a breach, there was a blog by a sort of skilled hacker that if a certain series of incidents occurred, you could possibly get in and obtain somebody's personally identifiable...

(CROSSTALK) ESHOO: But isn't that telling? Isn't that telling?

SEBELIUS: And we immediately corrected that problem, so there wasn't a -- it was a theoretical problem that was immediately fixed. I would tell you we are storing the minimum amount of data, because we think that's very important. The hub is not a data collector. It is actually using data centers at the IRS, at Homeland Security, at Social Security to verify information, but it stores none of that data, so we don't want to be.....

Secretary Sebelius is correct: I did not breach or exploit any of the vulnerabilities that I reported on my blog. And it is nice that she thinks I’m “sort of skilled” as a hacker, when I’m actually a highly-experienced software tester.

I identified a series of steps that could be easily automated to collect usernames, password reset codes, security questions, and email addresses from the system -- without any kind of authentication.

Attackers could use this information to go phishing. Exposing this information gives attackers sufficient information to gain trust and trick people into disclosing their security question answers.

If I were a malicious phisherman, I might send users email that directs them to a site masquerading as HealthCare.gov, and then ask victims to provide their security questions in order to revalidate their account. After collecting this information, I could then reset the password and access information the user provided to HealthCare.gov.

I found this issue last Thursday night (October 24th). I notified HealthCare.gov customer service immediately.  The next morning, I found someone who could help pass information about my discovery to people within HHS. CMS patched the most serious hole the same day, and made further changes on Monday before making a public statement about the issue.

“We are eliminating this theoretical vulnerability by preventing users from seeing the specific reset functionality when trying to reset their password... There is no public evidence that these design flaws were ever exploited to compromise user accounts." - Brian Cook, CMS Spokesman

While I am appalled that the issue existed in the first place, I applaud the quick response.

Monday night, after CMS publicly confirmed the fix, I took a quick look at the new "fixed" password reset functionality.

I saw a couple of positive changes:

1. The password reset code is no longer returned to the web browser. This closes the biggest hole. Exploiting weaknesses in the password reset system will now require that password reset codes be obtained by intercepting email, or some other mechanism.

2. The system asks for security question answers and a new password before submitting the request. This slows down manual security question guessing attempts, but will have little impact on an automated attack.

I also saw that many potential security issues still exist:

1. The system still confirms whether a username or email address exists in the error messages returned by the underlying services. Given that these are not public identifiers in the Insurance Marketplace, these should not be revealed. (As of 11/07, this still exists.)

2. The system still transmits both the username and password reset code via email. Email is generally not a secure means of communication. A more secure way to do this would be to send the user only half of the equation: the reset code; and then prompt the user for the username after they follow the reset link. (As of 11/08, this still exists.)

3. The password reset code still stays the same with each request (and is the same code used to initially activate the account). A more secure way to do this would be to change the code each time a password reset is requested or a password is reset; and in the case that a system contains sensitive information like this one does: put a time limit in which the reset code may be used. (As of 11/13, this appears to have been fixed. Reset codes are changing and old ones don't work.)

4. When requesting a password reset, the username and password reset code are still sent to 3rd party analytics companies. Even if these companies can be trusted to not misuse the data, this likely violates their terms of use and privacy policies. And if these 3rd parties aren't expecting personal information, they may not protect it as they would protect personal information. (As of 10/31, this appears to have been fixed.)

5. If any of the above (or other) issues lead to a username or password reset code being compromised, the security questions and email address associated with the account can still be retrieved from the system without authorization.

6. In the unfortunate event that an account is compromised:

1. An attacker can change the email address associated with the account without triggering notification of the email change to the user. Once this is done, other account information can be changed without notifying the owner of the account. (As of 11/05, this still exists.)

2. The personal information used to validate a user's identity is returned to the browser each time a user logs into a system. This data is both retained and returned to the browser when it should no longer be needed. Returning it to the browser each time a user logs into the system increases the potential damage should an account be compromised.  This data includes the personal information the account owner provided to verify their identity -- sufficient information to steal another’s identify. This also include all information entered into an insurance application for each person on the application (eg: names, DOB, SSNs, disability, pregnancy, finanical) and data retrieved from back-end systems (eg: employer, and income, and last paycheck details). (As of 11/13, the identify verification data is no longer returned to the browser; however, the personal information in the insurance application is still returned to the browser.)

Have you heard?

SLAVITT: Our systems don't hold data. They just transport data through it.

ROGERS: You don't have to hold it to protect it.

Both Secretary Kathleen Sebelius and Andy Slavitt, an executive VP at QSSI (the company tasked with fixing Healthcare.gov) have downplayed security concerns. They have suggested that personal information is not at risk because The Hub, the Healthcare.gov front end, does not store information; but rather, transports information. A system is only as secure as its weakest link. If front-end security is poor, then no amount of back-end security can protect information passing through the front end.

Even if Healthcare.gov doesn't store information, it returns personal information to the browser. As outlined above, the data I once provided to verify my identity is sent to my computer each time I login to Healthcare.gov -- long after the identity verification has been completed. This information includes name, address, date of birth, phone number, and Social Security Number. It also returns data retrieved from back-end systems; including, employer, income, last paycheck details, and more.

Much of the work required to exploit these vulnerabilities can be automated.

Many of these vulnerabilities are rather benign when considered individually. However, they quickly become more serious concerns if we consider how they may be combined and the exploits automated.

I have discovered several additional vulnerabilities while using the site.

1. The email validation system demonstrated the same flaw as the password reset system: It returned the activation code (which is the same as the password reset code) to the browser; enabling one to create an account using an email address they do not own. (As of 11/5, this appears to be fixed.)

2. My username and questionnaire answers were sent over the Internet without encryption under an error condition that also led to my profile information not being displayed. (As of 11/08, this still exists.)

3. The system returned Java stack traces to the browser; potentially revealing information about the internal workings or data of the system that could be exploited to find weaknesses in security. (As of 11/05, this still exists.)

4. The security questions ask for things that are likely to be known by one's friends, family, or ex -- and are the sorts of things many will post on Facebook or other social media. (This morning, HHS Secretary Sebelius referred to these questions as "personalized questions that can only be verified by you". They aren't.) (As of 11/08, this still exists.)

We can only speculate at what other security vulnerabilities might be found by someone willing to attempt to gain unauthorized access.

But wait, there's more!

UPDATE 10/31:

1. Healthcare.gov returns the username for an account when given a user's real name and email address. (This appeared to be fixed on 11/06, then reappeared on 11/08. As of 11/10, it appears to have been fixed again.)
2. Healthcare.gov returns the security questions for an account when given a username. (As of 11/11, this appears to have been fixed. I last say it on 11/08.)

No other authentication is required. Although this doesn't provide an attacker with the password reset code, it exposes information that should be kept private and provides sufficient information to make phishing relatively easy.

Conclusion

I am very happy that the most egregious issue was immediately fixed.

Others issues remain.

The vulnerabilities I've listed above are defects that should not make it to production.  It doesn't take a security expert or “super hacker” to exploit these vulnerabilities.

This is basic web security.  Most of these are the kinds of issues that competent web developers try to avoid; and in the rare case that they are created, are usually found by competent testers.

Rather than individual incompetence, these issues might suggest a fractured development team -- where the developers building the components don't know how they are used in the system and therefore do not have sufficient situational awareness to understand the security implications of their decisions. Still, someone has to assemble the components and the system as a whole should be tested for security. Given that I don't know what's going on within the project, I can only speculate. I have; however, seen enough to be concerned.

The volume of users, the nature of the data presumed in the system, and the political attention all contribute to making HealthCare.gov a target of interest to attackers -- of higher interest than the typical web site. This demands a higher standard. This requires that security be made a priority throughout design, implementation, testing, and monitoring of the system.

I am still concerned about Healthcare.gov security. It should concern all of us.

Is there a problem with Healthcare.gov?

I have discovered a number of issues with Healthcare.gov.  I have blogged details on some of these on my other blog, Is There A Problem Here?

These issues include, but are not limited to:

1) The site creates more cookie data than it will accept. It returns HTTP 400 errors (displaying blank screens to the user) when the cookie data it generates gets larger than parts of the site are configured to accept.

2) The site requires users create an account and verify identity and submit an application to get information about plan options. This creates a bottleneck that could have been avoided with different design.

3) The client-side Javascript code I've reviewed contains some errors and is overly complex -- complex in a way that adds overhead and risk that makes current understanding and future maintenance of the code unnecessarily difficult.

4) The site processed an application I did not submit -- and that I explicitly told it to not process.

5) There are so many obvious security flaws that I doubt they took security seriously. This gives me reason to be concerned about security of parts I can't see. Some of the security issues I've seen are:

• Personal data sent unsecured over HTTP
• Error messages that reveal the existence of usernames and email addresses in the system
• Stack traces returned to the browser that reveal information about the internal system components
• Usernames and password reset codes and questionnauire (not the application) answers sent to 3rd party analytics companies
• Password reset codes returned to the browser
• Email addresses associated with an account returned to the browser without authentication
• An email validation system that returns the info to needed validate an email address to the browser -- enabling people to create accounts using others' email addresses

If you want to see details, please visit my other blog at http://blog.isthereaproblemhere.com/search/label/Healthcare.gov

Ben Simo

What is Performance?

As we design and test for performance, let's look beyond speed. Let's look beyond basic stability. Let's look at the many facets of performance. Consider in which ways your system needs to perform. How fast does it need to be? How long does it need to keep running? Does it need to grow? Does it need to be available at all times? How much can we spend? Can we make it faster?

There are endless questions we could ask. Therefore, categorizing facets of performance and creating tests for each category can be helpful. However, let's not fail to look at the interaction between these categories.

OFAT (One Factor At a Time) testing (as exampled in the above performance testing checklist excerpt) often fails to provide information related to the interaction between the categories. Let's do some MFAT (Multiple Factors At a Time) testing and analysis. Let's look at the system as a whole. Let's mix it up. Let's consider how these facets interact. Let's create test scenarios that include multiple facets.

Time Machine: Devops, 45 Years Ago

"Two major inputs are required 

to provide computer service: 

equipment and manpower. 

For development, 

employees such as 

programmers and systems analysts are needed;

for operation, 

the services of managers, operators, etc., are essential. 

In practice the two activities usually coexist. 

Some sort of computer installation 

is required 

to check out development efforts. 

And almost every installation 

engages in continuing development 

as old systems are modified and new ones begun."

- William F. Sharpe, Economics of Computers, 1969

The above suggests that Devops is not at all a new concept. Forty-five years ago, computer rental cost slightly more than the Devops staff required to use it. Non-personnel operations cost typically rounded out the final third of the total cost. Today, hardware is cheap and people are expensive.

So, what did a Devops organization look like forty-five years ago?  The table below presents some data gathered in a "nation-wide census of data processing personnel" in 1967.

William F. Sharpe, Economics of Computers, 1969 

How do these wages add up in today's dollars?

Check out

The Co$t of Living.

Compared to my experience over the past couple decades, this seems to be heavy on management -- both in cost and percent of personnel. The roles of computer operator and librarian are pretty much gone. Programming and analyst roles have since blurred. Software has replaced many things that people used to do.  As the user base has moved from specialist operators to the general public, new testing and user experience design roles have evolved.

What else has changed?

What do you think devops will look like forty-five years into the future?

The Right Stuff?

Update: I haz new job. I am no longer seeking work. (12 June 2012)

You cannot be anything you want to be

-- but you can be a lot more of who you already are.

- Tom Rath, StrengthsFinder 2.0

Given that I resigned a couple weeks ago and am looking for new work, I've been asking myself a lot of questions about what I want to do next. Do I want to do detailed technical work? Do I want to lead people? Do I want to consult? Do I want to be someone's employee, or do I want to do short term contract work? Do I want to stay in Arizona, go back to Colorado, or go somewhere new? I've not yet decided. I am considering a number of options.


What I do know is that I want to find work that is a good match for my strengths and mindset. When looking for new work for myself, and when I've been involved in hiring, matching strengths and mindset has been more important to me than matching specific technical skills. People who are enabled to exercise their strengths and philosophy tend to have the intrinsic motivation needed to update their skills as each situation demands.

Rightshifting:

Improving the effectiveness 

of knowledge-work businesses.

I am intrigued by the Rightshifting ideas coming from Bob Marshall, the Flowchain Sensei. The basic idea of the Rightshifting model is that the curve shown below indicates that the majority of organizations (peak of the curve) are ineffective compared to the few effective organizations on the right side of the curve. While organizations to the right of Bob's model tend to better match my personal mindset, finding work with such organizations is difficult -- there aren't many of them. My experience suggests that organizations further to the left can better benefit from what I have to offer than organizations to the right. However, if such organizations have no desire to improve their effectiveness, I am likely to get frustrated. Therefore, one more question I am asking myself is whether I prefer to be more of a hero in the midst of analytical dysfunction or more of a team member in a synergistic organization. I suspect my place is somewhere in the middle -- with an organization that is shifting right.

Side Note: One way in which I think I may disagree with Bob Marshall's model is the assertion that the chaotic organizations on the left are less productive than the analytic organizations to their right. I suspect many of the majority (the analytical organizations) are just better at measuring (or pretending to measure) their productivity than the organizations to their left. What we label chaos is often order we don't yet understand well enough to describe. I'm also not convinced that organizations need to take a trip through the dehumanizing methods of many analytical organizations in order to become more productive.

I recently watched a video in which Bob Marshall recommended the Strengths Finder book and the associated online assessment as a useful tool in discovering your strengths. I am typically skeptical of such assessments. I fear they tend to provoke the Forer effect in which people tend to demonstrate a confirmation bias and attribute accuracy to generalized information about themselves when it is presented as being specifically tailored for them. However, I decided to give Strengths Finder a try. Despite my reservations and desire to critically evaluate the timed questionnaire as I completed it, I found the resulting top "themes" to be a surprisingly accurate description of my strengths. Perhaps this only demonstrates that even a skeptic like me can succumb to the Forer effect.

According to Strengths Finder, my top five strengths are:

1. Ideation
2. Command
3. Activator
4. Individualization
5. Relator

Whether this is based on a meaningful algorithm or is total BS, I found the results useful. (All models are wrong, but some are useful.)  These five strengths seem to example what I believe can make a good context-driven tester; but then, I'm biased. :)

So, back to what do I want in my next job or independent business venture: I want to exercise my strengths in an environment where I don't have to suffer cognitive dissonance in order to serve my employer or clients.

In thinking about this, I created a mind map of the strengths from Strengths Finder and things I like, as well as things I don't like. Following up on my desire to find a mindset match over simply matching skills, I've decided to ignore typical job hunting advice and share this.

Think I might be a match for your organization? Hire me. :)

Target Tested ✔

Computers empower us to create and manipulate things in ways that don't produce quite what we intended. This requires testing for what we don't expect.

There is a problem with this advertisement from Target Australia. Can you spot it? 

Do the math!

Sometimes numbers that make no sense are presented in ways that seem to make sense... or not. :)
 

uTest Interview

uTest invited me to be interviewed for Testing the Limits series this month.  They have graciously referred to me as "one of the most insightful and entertaining testers in the business". Please give the interview a read.  And if you have additional questions, please send them my way.

• Part I. Worst bug, testing philosophy, defensive pessimism, certifications
• Part II. Trusting automation, why Quality Assurance is a bad title, overstructured managemenet
• Part III. How I got into testing, what would I be doing if there were no such thing as software

What Sesame Street taught me about problem solving

A checklist for buying a new computer

Buying a computer has changed significantly in the past 25 years. Or not?

From Melbourne's The Age, 18 June 1984

Some features that are really important to me:

• on-off switching (the most important feature)
• security against blow-up (because I fear burns and shrapnel from exploding computers)
• position of reset key (a reset key next to the shift key would be a pain)
• big yellow bus (aren't computers good for education?)
• radiation shielding (don't want to be sterilized while playing Pole Position)
• movable screen (I might want to take the screen home with me)
• ease of plug-in ROM installation (cartridges are better than tapes)
• recognize commands in u/l case (I expect the computer to understand me -- EVEN WHEN I YELL AT IT!)
• hard disc option (I prefer to save data to tape, but a hard disc would be a nice option)
• stand-alone capability of slaves (I want slaves that can work without being micro-managed)
• octave range (I need a computer that can sing soprano)
• RS-232C port (In case I want to interface it to my lawn sprinkler system)
• Paddles (to spank the computer when it misbehaves)
• key-word dictionary (to understand this check list)
• country of origin (can I get a machine made in the USA?)
• dynamic debugging tool (certainly don't want a static debugging tool)
• Logo (because I just love that little turtle)