When zero-day vulnerabilities hit, it often feels like we have zero time to figure out the risk to our systems.
Take the recent Log4Shell vulnerability as an example. By the way, Rundeck is a Java-based application, and older versions were subject to this vulnerability. If you have not yet updated to Rundeck 3.4.8 or later, please do so now.
In this blog I will show you how Rundeck can be a force multiplier in helping teams quickly identify systems that need attention and take action where needed.
Here I have a lab environment that we’ll use to walk through the thought process of chasing down Log4Shell vulnerable systems.
In our lab environment, there are two Linux nodes. We aren’t sure if there are any vulnerable packages on these systems, so let’s find a way to scan them.
Our team did some research and settled on scanning with a tool provided by LunaSec. (Note: This isn’t necessarily an endorsement or recommendation. This is one of MANY tools that could be used. We are simply using this for demonstration purposes only).
Since this tool is new to us, we’ll start using Rundeck’sAdHoc Commands feature to get the binary onto a single Linux box. In our Commands windows, we’ll run these against a single node. (NODE1)
wget https://github.com/lunasec-io/lunasec/releases/download/v1.3.1-log4shell/log4shell_1.3.1-log4shell_Linux_x86_64
should get it onto our system
Let’s rename that to something easier to call
mv log4shell_1.3.1-log4shell_Linux_x86_64 log4shell
… and make it executable
sudo chmod 755 log4shell
Now let’s test it using the syntax from the Lunasec docs.
sudo log4shell s /
In our video demo, it was able to find some planted vulnerable packages on Node 1.
So in quick order, we’ve used AdHoc Commands to install a scanning tool and identify a concern on our first node.
The next step would be to take these same steps and turn them into jobs which can be reused across a wider set of nodes and in the future. We can view all the commands we ran in the Activity tab and combine those into jobs with additional logic as needed.
We split the work into two jobs. One installs the log4shell command in /usr/bin so it can be run by anyone on the node.
The main script will check for vulnerabilities in any specified directory using the log4shell command. If the command is missing or hasn’t been installed yet, this job will call the installation job to run the install script, then it will complete the scan.
The script will fail a Node if the script returns vulnerabilities, and pass the Node if it’s safe. Running it, we can see Node1 still fails due to our planted vulnerable Log4j files. But Node2 is healthy. Once we clean up Node 1, the job will pass successfully.
This automated diagnostic of which servers are vulnerable provides a first step towards remediation. How to fix each one depends on its own unique scenario, but of course Rundeck could help, too. For now, we could schedule this job to regularly scan any hundreds of systems to ensure they are healthy. If the job ever fails, use a Job Notification to open an incident in PagerDuty and notify the proper individuals.
So…go forth and automate!
Below is code for the two jobs used in the video demo*
As a followup bonus, check out this learning article that can be used with a Rundeck Health Check to look for nodes that might have the vulnerability and highlight that in the nodes listing.
*These jobs are provided as examples only to illustrate a design pattern and are not intended to provide security for any environment. Since this was recorded LunaSec may have issued newer versions of the tool so be sure to adjust the jobs for the latest version.
You might also notice that when scanning Rundeck 3.4.8 (you did upgrade to the latest version right!!??) the tool identifies a vulnerability in the Log4j 2.16 package as 2.14. We have confirmed this is a false positive in Log4Shell and they are working on patching it. Be assured Rundeck 3.4.8 is patched and protected from CVE-2021-44228 and CVE-2021-45046. There is also another CVE from 2019 not related to the recent Log4Shell CVEs.
Installation Job
- defaultTab: nodes
description: ''
executionEnabled: true
loglevel: INFO
name: Scan Directory with Log4Shell
nodeFilterEditable: true
nodefilters:
dispatch:
excludePrecedence: true
keepgoing: true
rankOrder: ascending
successOnEmptyNodeFilter: false
threadcount: '1'
filter: .*
nodesSelectedByDefault: true
options:
- label: Directory to Scan
name: DirectoryPath
value: /
plugins:
ExecutionLifecycle: {}
scheduleEnabled: true
schedules: []
sequence:
commands:
- description: Check Log4Shell Version
errorhandler:
jobref:
group: ''
name: Install Log4Shell
nodeStep: 'true'
useName: 'true'
uuid: c89d6ff5-18a8-4165-838d-e7c1fb693c3d
keepgoingOnSuccess: true
exec: sudo log4shell -v
- interpreterArgsQuoted: false
script: |-
if log4shell s --json @option.DirectoryPath@ 2>&1 | grep -q -E '(44228|45046)'
then
echo "Found vulnerable to Log4Shell"
log4shell s @option.DirectoryPath@ 2>&1
exit 1;
fi
scriptInterpreter: sudo
keepgoing: false
strategy: node-firstScanning Job
- defaultTab: nodes
description: ''
executionEnabled: true
loglevel: INFO
name: Scan Directory with Log4Shell
nodeFilterEditable: true
nodefilters:
dispatch:
excludePrecedence: true
keepgoing: true
rankOrder: ascending
successOnEmptyNodeFilter: false
threadcount: '1'
filter: .*
nodesSelectedByDefault: true
options:
- label: Directory to Scan
name: DirectoryPath
value: /
plugins:
ExecutionLifecycle: {}
scheduleEnabled: true
schedules: []
sequence:
commands:
- description: Check Log4Shell Version
errorhandler:
jobref:
group: ''
name: Install Log4Shell
nodeStep: 'true'
useName: 'true'
uuid: c89d6ff5-18a8-4165-838d-e7c1fb693c3d
keepgoingOnSuccess: true
exec: sudo log4shell -v
- interpreterArgsQuoted: false
script: |-
if log4shell s --json @option.DirectoryPath@ 2>&1 | grep -q -E '(44228|45046)'
then
echo "Found vulnerable to Log4Shell"
log4shell s @option.DirectoryPath@ 2>&1
exit 1;
fi
scriptInterpreter: sudo
keepgoing: false
strategy: node-firstIf you don’t already have a PagerDuty account, sign up for a free Guided Rundeck Trial today
