Defensive Go back to all

Blog

Linux Purple Team Exercises - Use Case Story

- By Leszek Miś

This week, I completed a 10-day external project called Linux Purple Team Exercises. All interactions with the customer, comprised of members from the offensive, detection engineering, and CSIRT squads, took place on-site at the company's headquarters in Frankfurt. In this post, I would like to show you how I approached planning, preparation, and execution, and what outcomes we achieved together.

What is Purple Teaming?

Personally, that's how I see it - Purple Team Exercises is a collaborative approach that combines elements of red teaming (adversary emulation) and blue teaming (defense simulation). The red team works closely with the blue team (internal security defenders) to share insights and improve the overall security posture.

Adversary emulation involves simulating the tactics, techniques, and procedures (TTPs) of real-world threat actors to assess an organization's security defenses and incident response capabilities. This process aims to replicate the behaviors of real adversaries, enabling security teams to understand their vulnerabilities and weaknesses better. This proactive approach helps organizations enhance their security posture by identifying and addressing potential gaps in their defenses.

Adversary emulation is not a one-time activity; it's an ongoing process. Organizations can use the insights gained from these simulations to continuously improve their security controls, detection capabilities, and incident response procedures.

Adversary emulation helps evaluate the effectiveness of security controls, including intrusion detection systems, firewalls, endpoint protection, and other defensive measures. This assessment is critical for identifying and addressing weaknesses in the security infrastructure.

Planning and preparation:

A private, isolated infrastructure was needed. For this purpose, a dedicated security group was created within AWS, within which we launched two dedicated Alma 9 Linux VMs, as well as an internal Kali Linux machine. Additionally, a dedicated VPS was deployed with a public IP address and assigned domain names (egress testing/C2 callbacks on different layers). Security tooling was installed within each Alma Linux instance, including EDR, SIEM, and DFIR agents actually used in production. Each of these components was connected to the true production detection stack, allowing us to track individual artifacts of launched offensive techniques live. The AWS Security group's network configuration was a 1:1 replica with those used in the production environment, i.e., no direct access to the Internet, proxy for egress, internal DNS server, etc. Alma Linux machines had a set of scripts deploying vulnerable network services, as well as several local misconfigurations, vulnerabilities in the kernel, and within userspace binaries. I believe it is worth showing full attack paths in such PT exercises, because this way we can get the full context of the attackers' flow and, consequently, gain visibility based on alerts and telemetry from the entire chain. In addition to initial access via remote exploitation and obtaining a shell, we also had the ability to run individual techniques using regular user and root privileges in line with the "Assume Breach" approach.

Emulating real threat actors provides a realistic context for assessing security defenses. This approach goes beyond simple vulnerability scanning by considering how attackers might leverage multiple vulnerabilities and tactics in a coordinated manner.

Security teams create custom attack paths that replicate the steps a real attacker might take to compromise systems and achieve their objectives. These attack paths often involve a series of simulated steps, from initial access to data exfiltration.

Range of Linux offensive techniques:

You are correct in assuming that we have divided the techniques used into tactics based on the MITRE Attack Framework. The main source of offensive techniques in my case was the EDRmetry Linux Matrix, and I admit that I did not understand how much value this approach really provides. Fueled by continuous research into emerging threats, EDRmetry Linux Matrix serves as a dynamic hub for cutting-edge offensive Linux expertise. Know your enemy through hands-on experience - this is the motto that has always accompanied me.

Generally, EDRmetry is a central knowledge base that allows you to easily search for a given technique by keywords, and the result is a dedicated EDR-TXXX containing pieces of code and commands to be executed in a step-by-step format. 

At the moment, the number of ready-for-chaining dedicated offensive techniques included in the EDRmetry library is almost 500 and is constantly growing. You can really purple-test a lot and have a huge amount of low-level discussions supported by examples with a tool like this. I must admit, building EDRmetry was a real success. It also works great for workshops, presentations, and live training sessions, and I've done quite a few of those myself. There are a lot of them ahead of me, too:

The flow:

Okay, you may ask what it looked like in real life? I personally chose the chain or technique, but we can also determine the execution together. Every contextual trigger should be visible in some way within alerting, live telemetry, or triage. Sometimes I immediately discuss what has just been launched, and sometimes I only provide a “magic” keyword, based on which the team could start their investigation. The most interesting part was the actual support of detection analysis and pivoting between different data sources. If you get a good team, as I did, most of the discussions and brainstorming will take place on the syscall level - very inspiring but also very challenging. It was also great to be able to leverage my own research from Linux Attack, Detection and Forensics v2.0 - Hands-on Purple Teaming Playbook, drawing on detection and telemetry from Purplelabs and my blue team stack, which includes Falco, Kunai, Tetragon, LKRG, Linux_IR_Scripts, Sandfly, Elastic Security, Velociraptor, and OSQuery. Confirming what can be detected and comparing it with the customer's results always provides added value and understanding of what we don't have and what is actually achievable.

Here is a list of some questions/comments that appeared during the Linux PT Exec project:

  • What are the child processes spawned from the parent Java process?
  • Why is the PID/PPID invisible? Have you checked the mounts carefully?
  • This proc has no environment variables attached.
  • Did you catch a file_create event in the Document Root?
  • Why did this dropped binary execute init_module/finit_module?
  • I see a potential privilege escalation in ps forest view
  • Oh, I see execve from /usr/bin/…x
  • No wget, no curl? But still, the implant was deployed over the network. Ok, ok, I see connect/send_data events from bash or openssl. 
  • Let’s dump process heap and stack memory to reveal defined bash functions 
  • Sliver / Mythic implant detected. Ok. What about curlrevshell approach?
  • I see you have modified core_pattern from within the container, but why?
  • I’ve found a Java process class file download event from the pod.
  • Ohh, this process masquerades as ps aux ;]
  • You said I need a http_proxy? Look, I have a full C2 on egress to S3 without proxy.
  • What happens if we drop via iptables all IPs that EDR/SIEM connects to?
  • You have modified sshd_config, and I see /root/.ssh/authorized_keys file_create events. Why does sshd create files in $HOME on its own?
  • So, a local socket file can be forwarded over the network as an open port? Chisel+socat love
  • I see some bpf_prog_load events, but I don't see anything interesting on bpftool output.
  • Do you see an event corresponding to the direct memory access of the PID?
  • Do you see any injection symptoms, like ptrace anyone?
  • WTF, why does this kernel thread connect to IP:Port? 
  • “is installing a program with bpf_probe_write_user helper that may corrupt user memory”
  • Fileless exec is not always memfd_create, but it works great when FS is noexec
  • I just executed in memory BOF. mprotect_exec anyone?
  • Can you notice many requests to http://169.254.169.254/latest/meta-data/ executed via your app daemon?
  • We just got an alert from the SOC on guard. Apparently, long, exotic, and base64 encoded DNS req/resp have been seen on the network => correct, I can execute commands blindly/one-way communication only, because of your DNS setup.
  • Do you have visibility for call_usermodehelper? Can you block it somehow? I see what happens on UMH via LRKG, and I can block its execution via Tetragon. What about you guys?
  • In case you want to reveal hidden pids, just do: echo t > /proc/sysrq-trigger
  • info.event.name=io_uring_sqe is your friend
  • 7fff9f7fd000-7fff9fffd000 rwxp 00000000 00:00 0 
  • All your implant has to support is SOCKS proxy and BOF loading
  • This actuator endpoint should not be here :> Can you spot mem dump?
  • OK, we got an interesting bpf_socket_filter event. 
  • Unknown binary execution from /dev/shm is always suspicious

These are just a few interesting threads, and there were many more. Each thread was subject to discussion and manual inspection of the corresponding detection logic. Really amazing experience!

Outcomes:

Thanks to the PT exercises, many blind spots were detected, which were often patched live.  The beauty of custom setup - you don't have to wait weeks to be provided with a silly answer from your vendor. The team was equipped with new offensive knowledge, crucial for a better understanding of the actual range of Linux threats and attacker behavior. The exercises verified the actual status of achieved visibility and ultimately led to faster incident response times. Only contextual execution and a broad perspective count. Good tools and automated alert flow are important, but equally important are the basics of Linux operation and architecture. This knowledge is necessary to determine whether an event should be escalated or simply derived from the baseline. And in large infrastructure, it is not that simple. In general, understanding what to expect from modern EDR/SIEM/DFIR systems, what Linux EDR consists of, and what level of threat detection, visibility, and response it offers is crucial. Purple Teaming for life!

The future:

Given that AI is helping to find more and more critical vulnerabilities, I assume that the number of actual breaches will only increase - at least for some time. It seems to me that defensive security, DFIR, and low-level Linux hardening are even more important than ever. While everything is naturally bypassable, we should focus on continuously validating detection coverage and achieving a level of process automation (SOAR-ish setups?) that ultimately shortens the time to detection/time to response. Remember that even the smallest alert in a specific context matters, as it determines whether or not we initiate a response. I also believe that SOC/CSIRT teams should spend more time on hands-on Linux practice, as I have seen over the years. Of course, this should mainly focus on Red vs. Blue, but also on a basic understanding of Linux architecture, individual subsystems, and the behavioral characteristics of their workloads. Not an easy task at scale, though.

 

Manual, contextual Purple teaming, including knowledge transfer, is very valuable, but in the age of AI, we need to power up AI in this area too. I would like SOC/CSIRT/Detection Eng teams to be able to launch any Linux offensive techniques without my assistance as a Purple Team Operator in a fully automatic, understandable, safe, and controlled manner, but above all, maintaining context and providing live interaction with, ex. whatever C2 connection that has been spawned or local LPE. Something like atomics-ng. That’s the reason why I work hard on EDRmetry Pulse - AI-Assisted Offensive Linux Operator/Contextual TTP-based Telemetry Generator. Stay tuned!

LINKS:

Categories: