Security Design Flaws and Serverless Billing

I've been doing a lot of reflecting on a topic. Fundamental design flaws. They are the most powerful type of exploit to stumble into, and the cloud community just had one that was "fixed" within 24hr.

These are the type of vulnerabilities that are incredibly expensive to fix. Finding them is hard. Disclosing them is harder.

These vulnerabilities are the now classics that represent a fundamental shift in some industries. Spectre. Meltdown. Heartbleed. Kaminksy's DNS work.

A common trend between them is uncovering a design flaw in a technology that others build on and have been in use for a sufficiently long time. Because of this, they affected millions to billions of devices. They also tend to be bugs that exploit side-channels.

If you are searching for design flaws. I suggest looking at technologies which are relied on. Then I would suggest looking for threat models and the promises made about the technology in the marketing.

Learn about how the technology was originally designed and if that design was changed over the years. See if the marketing actually matches a claim that can be proven.

Big bold claims are where the dragons lay and also why anyone in their right mind would never utter the words "unhackable".

Heartbleed had a side-channel. Spectre & Meltdown were side channels. Kaminsky's DNS cache poisoning used knowledge of the base properties/limitations of networking.

All of these problems were "by protocol design" or "working as intended" which is the phrase every security professional sighs at.

In the AWS cloud security space we had a run in with a nasty denial-of-wallet which I would argue was a design flaw in their billing model.

When you charge for serverless, it's usually done "per request". It seems reasonable at first glance. Pay for what you use, if you don't use it, you don't pay!

Some actions are more expensive. Like actions PUT, COPY, POST, LIST which all require AWS to use computational things on your behalf they charge $0.005/thousand (1 million is $5) in most cases. For retrieval, it’s even cheaper at $0.0004/thousand (or 1 million is $0.40) then there's a charge for the amount of data being sent out of AWS. Seems reasonable.

But let’s talk edge cases. What if the request is a 5XX internal server error? Do you expect AWS to make you pay for something that's their fault? Nah. They agree, that's on their side of the shared responsibility model, that shouldn't be on the customer.

Now what about 4XX or 3XX? Is that on us? Yes, AWS says that is on us because they had to use compute resources. After all, if you were hosting your own server, you'd be spending money too. Luckily, if something like a 403 unauthorized happens, there's not really any data transferred out anyway. This all seems reasonable.

So what's the problem?

What if we have a non-public S3 bucket. And someone knows the bucket name, has an AWS account, and is making a PUT or POST requests to your bucket causing a 403 unauthorized error? You own the resource, you pay for it. Except now, it’s $5 per million instead of $0.40 per million.

How fast can I make those requests? Let's say an easy 100 request/second, which is about $43 a day. That's $1296/mo with standard AWS pricing. Now S3 Glacier Deep Archive is 10x more expensive, which would be $12,960/mo.

That is just assuming you are using a single computer, which would almost certainly not be the case.

How do we stop it? Well...YOU can't if the attacker knows the bucket name. AWS S3 didn't design resources so that you hide S3 bucket names or declare they were sensitive and should be hidden.

They're designed to be globally unique and encouraged to be human-readable. Also, because you want other internal AWS services to interact with your buckets, they have to be reachable by the AWS backend, so AWS can determine authorization. That's a special exception worth noting here, but one that doesn't exist in our side of the shared responsibility model.

The fundamental problem here is that an attacker is weaponizing the AWS backend infrastructure against you, while you are responsible for the bill. To make it worse, you cannot control the fix with tools on your side of the shared responsibility model. S3 buckets were never designed to be hidden and weren't designed to be entirely unreachable by the AWS backend.

But Lizzie, what if we use CloudFront or a WAF and hide the bucket name? Fair, for public buckets, but this works for private buckets too. If an attacker finds the bucket name, you're in the same spot. Also, again, bucket names aren't considered sensitive, so now we have to redesign everything with that in mind.

Once a bucket name is known, you just use AWS's backend directly. There were only two real solutions here, and it was either AWS updating their billing design for this or to implement something on their side of the shared responsibility model.

They did a 24hr turnaround and are changing the billing model at a company of their size, which is insane.

It speaks to quick risk evaluation and being able to pull a fire alarm and have staff react. All of these are good things you want to see from a cloud provider.

It also shows that the money involved from these requests wasn’t even close to the money involved with implementing a redesign of core cloud compute architectures.

Earlier, I gave some non-cloud examples where they had to completely rearchitect and backwards patch, the cost of all of that was completely enormous and needed coordination with lots of people. Imagine if the researchers in those situations didn’t stick to their guns and back down when met with “working as intended.”

Part of the reason I have been thinking about designs and black swan events is because my research partner and I are releasing some research attacking a fundamental design.

It's been an uphill battle from the start, and now suddenly it’s being taken seriously behind the scenes once it had more momentum.

I hope this post was helpful, or at the very least, interesting.