<section class="blog-post-content"><p>In this blog post, we will walk you through the <a href="https://github.com/aws-samples/ec2-forensic-module-factory" target="_blank" rel="noopener noreferrer">EC2 forensic module factory solution</a> to deploy automation to build forensic kernel modules that are required for <a href="https://aws.amazon.com/ec2/" target="_blank" rel="noopener noreferrer">Amazon Elastic Compute Cloud (Amazon EC2)</a> incident response automation.</p><p>When an EC2 instance is suspected to have been compromised, it’s strongly recommended to investigate what happened to the instance. You should look for activities such as:</p><ul><li>Open network connections</li><li>List of running processes</li><li>Processes that contain injected code</li><li>Memory-resident infections</li><li>Other forensic artifacts</li></ul><p>When an EC2 instance is compromised, it’s important to take action as quickly as possible. Before you shut down the EC2 instance, you first need to capture the contents of its <a href="https://docs.aws.amazon.com/whitepapers/latest/aws-security-incident-response-guide/capturing-volatile-data.html" target="_blank" rel="noopener noreferrer">volatile memory</a> (RAM) in a <em>memory dump</em> because it contains the instance’s in-progress operations. This is key in determining the root cause of compromise.</p><p>In order to capture volatile memory in Linux, you can use a tool like <a href="https://github.com/504ensicsLabs/LiME" target="_blank" rel="noopener noreferrer">Linux Memory Extractor (LiME)</a>. This requires you to have the kernel modules that are specific to the kernel version of the instance for which you want to capture volatile memory. We also recommend that you limit the actions you take on the instance where you are trying to capture the volatile memory in order to minimize the set of artifacts created as part of the capture process, so you need a method to build the tools for capturing volatile memory outside the instance under investigation. After you capture the volatile memory, you can use a tool like <a href="https://github.com/volatilityfoundation/volatility" target="_blank" rel="noopener noreferrer">Volatility2</a> to analyze it in a dedicated forensics environment. You can use tools like LiME and Volatility2 on EC2 instances that use x86, x64, and Graviton <a href="https://aws.amazon.com/ec2/instance-types/" target="_blank" rel="noopener noreferrer">instance types</a>.</p><h2>Prerequisites</h2><p>This solution has the following prerequisites:</p><h2>Solution overview</h2><p>The EC2 forensic module factory solution consists of the following resources:</p><ul><li>One <a href="https://aws.amazon.com/step-functions/" target="_blank" rel="noopener noreferrer">AWS Step Functions</a> workflow</li><li>Two <a href="https://aws.amazon.com/lambda/" target="_blank" rel="noopener noreferrer">AWS Lambda</a> functions</li><li>One <a href="https://aws.amazon.com/systems-manager/" target="_blank" rel="noopener noreferrer">AWS Systems Manager</a> document (SSM document)<blockquote><p><strong>Important</strong>: The SSM document clones the <a href="https://github.com/504ensicsLabs/LiME" target="_blank" rel="noopener noreferrer">LiME</a> and <a href="https://github.com/volatilityfoundation/volatility" target="_blank" rel="noopener noreferrer">Volatility2</a> GitHub repositories, and these tools use version 2.0 of the <a href="https://www.gnu.org/licenses/licenses.html" target="_blank" rel="noopener noreferrer">GNU General Public License</a>. This SSM document can be updated to include your preferred tools, like <a href="https://github.com/NateBrune/fmem" target="_blank" rel="noopener noreferrer">fmem</a> or <a href="https://github.com/volatilityfoundation/volatility3" target="_blank" rel="noopener noreferrer">Volatility3</a>, for forensic analysis and capture.</p></blockquote></li><li>One <a href="https://aws.amazon.com/s3/" target="_blank" rel="noopener noreferrer">Amazon Simple Storage Service (Amazon S3)</a> bucket</li><li>One <a href="https://aws.amazon.com/vpc/" target="_blank" rel="noopener noreferrer">Amazon Virtual Private Cloud (Amazon VPC)</a></li><li>One security group for the EC2 instance that is provisioned during the automation</li><li>The solution uses the following VPC endpoints for AWS services:<ul><li>ec2_endpoint</li><li>ec2_msg_endpoint</li><li>kms_endpoint</li><li>ssm_endpoint</li><li>ssm_msg_endpoint</li><li>s3_endpoint</li></ul></li></ul><p>Figure 1 shows an overview of the EC2 forensic module factory solution workflow.</p><div id="attachment_27359" class="wp-caption aligncenter c5"><img aria-describedby="caption-attachment-27359" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2022/09/23/img1-1-1024x704.png" alt="Figure 1: Automation to build forensic kernel modules for an Amazon Linux EC2 instance" width="760" class="size-large wp-image-27359 c4" /><p id="caption-attachment-27359" class="wp-caption-text">Figure 1: Automation to build forensic kernel modules for an Amazon Linux EC2 instance</p></div><p>The EC2 forensic module factory solution workflow in Figure 1 includes the following numbered steps:</p><ol><li>A Step Functions workflow is started, which creates a <a href="https://docs.aws.amazon.com/step-functions/latest/dg/connect-to-resource.html#connect-wait-token" target="_blank" rel="noopener noreferrer">Step Functions task token</a> and invokes the first Lambda function, createEC2module, to create EC2 forensic modules.<ol class="c6"><li>A Step Functions task token is used to allow long-running processes to complete and to avoid a Lambda timeout error. The createEC2module function runs for approximately 9 minutes. The run time for the function can vary depending on any customizations to the createEC2module function or the SSM document.</li></ol></li><li>The createEC2module function launches an EC2 instance based on the Amazon Machine Image (AMI) provided.</li><li>Once the EC2 instance is running, an SSM document is run, which includes the following steps:<ol class="c6"><li>If a specific kernel version is provided in step 1, this kernel version will be installed on the EC2 instance. If no kernel version is provided, the default kernel version on the EC2 instance will be used to create the modules.</li><li>If a specific kernel version was selected and installed, the system is rebooted to use this kernel version.</li><li>The prerequisite build tools are installed, as well as the LiME and Volatility2 packages.</li><li>The LiME kernel module and the Volatility2 profile are built.</li></ol></li><li>The kernel modules for LiME and Volatility2 are put into the S3 bucket.</li><li>Upon completion, the Step Functions task token is sent to the Step Functions workflow to invoke the second cleanupEC2module Lambda function to terminate the EC2 instance that was launched in step 2.</li></ol><h2>Solution deployment</h2><p>You can deploy the EC2 forensic module factory solution by using either the <a href="https://aws.amazon.com/console/" target="_blank" rel="noopener noreferrer">AWS Management Console</a> or the <a href="https://aws.amazon.com/cdk/" target="_blank" rel="noopener noreferrer">AWS Cloud Development Kit (AWS CDK)</a>.</p><h3>Option 1: Deploy the solution with AWS CloudFormation (console)</h3><p>Sign in to your preferred security tooling account in the <a href="https://console.aws.amazon.com/" target="_blank" rel="noopener noreferrer">AWS Management Console</a>, and choose the following <strong>Launch Stack</strong> button to open the AWS CloudFormation console pre-loaded with the template for this solution. It will take approximately 10 minutes for the CloudFormation stack to complete.</p><p><a href="https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=ec2forensicmodulefactory&amp;templateURL=https://awsiammedia.s3.amazonaws.com/public/sample/1409-ec2-forensic-kernel-module-build/ec2_module_factory_cfn.yaml" rel="noopener noreferrer" target="_blank"><img src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2019/06/05/launch-stack-button.png" alt="Select this image to open a link that starts building the CloudFormation stack" width="190" height="36" class="aligncenter size-full wp-image-10149" /></a></p><h3>Option 2: Deploy the solution by using the AWS CDK</h3><p>You can find the latest code for the EC2 forensic module factory solution in the <a href="https://github.com/aws-samples/ec2-forensic-module-factory" target="_blank" rel="noopener noreferrer">ec2-forensic-module-factory GitHub repository</a>, where you can also contribute to the sample code. For instructions and more information on using the AWS CDK, see <a href="https://aws.amazon.com/getting-started/guides/setup-cdk/" target="_blank" rel="noopener noreferrer">Get Started with AWS CDK</a>.</p><p><strong>To deploy the solution by using the AWS CDK</strong></p><ol><li>To build the app when navigating to the project’s root folder, use the following commands.<code class="c7">npm install -g aws-cdknpm install</code></li><li>Run the following commands in your terminal while authenticated in your preferred security tooling AWS account. Be sure to replace <INSERT_AWS_ACCOUNT> with your account number, and replace <INSERT_REGION> with the AWS Region that you want the solution deployed to.<code class="c7">cdk bootstrap aws://<INSERT_AWS_ACCOUNT>/<INSERT_REGION>cdk deploy</code></li></ol><h2>Run the solution to build forensic kernel objects</h2><p>Now that you’ve deployed the EC2 forensic module factory solution, you need to invoke the Step Functions workflow in order to create the forensic kernel objects. The following is an example of manually invoking the workflow, to help you understand what actions are being performed. These actions can also be integrated and automated with an EC2 incident response solution.</p><p><strong>To manually invoke the workflow to create the forensic kernel objects (console)</strong></p><ol><li>In the <a href="https://console.aws.amazon.com/" target="_blank" rel="noopener noreferrer">AWS Management Console</a>, sign in to the account where the solution was deployed.</li><li>In the <a href="https://us-east-1.console.aws.amazon.com/states/home?region=us-east-1#/statemachines" target="_blank" rel="noopener noreferrer">AWS Step Functions console</a>, select the state machine named <strong>create_ec2_volatile_memory_modules</strong>.</li><li>Choose <strong>Start execution</strong>.</li><li>At the input prompt, enter the following JSON values.<code class="c7">{"AMI_ID": "ami-0022f774911c1d690","kernelversion":"kernel-4.14.104-95.84.amzn2.x86_64"}</code></li><li>Choose <strong>Start execution</strong> to start the workflow, as shown in Figure 2.<div id="attachment_27363" class="wp-caption aligncenter c8"><img aria-describedby="caption-attachment-27363" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2022/09/23/img2-3-1024x309.png" alt="Figure 2: Step Functions step input example to build custom kernel version using Amazon Linux 2 AMI ID" width="700" class="size-large wp-image-27363 c4" /><p id="caption-attachment-27363" class="wp-caption-text">Figure 2: Step Functions step input example to build custom kernel version using Amazon Linux 2 AMI ID</p></div></li></ol><h2>Workflow progress</h2><p>You can use the AWS Management Console to follow the progress of the Step Functions workflow. If the workflow is successful, you should see the image when you view the status of the Step Functions workflow, as shown in Figure 3.</p><div id="attachment_27364" class="wp-caption aligncenter c9"><img aria-describedby="caption-attachment-27364" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2022/09/23/img3-3.png" alt="Figure 3: Step Functions workflow success example" width="450" class="size-full wp-image-27364 c4" /><p id="caption-attachment-27364" class="wp-caption-text">Figure 3: Step Functions workflow success example</p></div><blockquote><p><strong>Note</strong>: The Step Functions workflow run time depends on the commands that are being run in the SSM document. The example SSM document included in this post runs for approximately 9 minutes. For information about possible Step Functions errors, see <a href="https://docs.aws.amazon.com/step-functions/latest/dg/concepts-error-handling.html" target="_blank" rel="noopener noreferrer">Error handling in Step Functions</a>.</p></blockquote><p><strong>To verify that the artifacts are built</strong></p><ol><li>After the Step Functions workflow has successfully completed, go to the S3 bucket that was provisioned in the EC2 forensic module factory solution.</li><li>Look for two prefixes in the bucket for LiME and Volatility2, as shown in Figure 4.<div id="attachment_27365" class="wp-caption aligncenter c10"><img aria-describedby="caption-attachment-27365" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2022/09/23/img4-1.png" alt="Figure 4: S3 bucket prefix for forensic kernel modules" width="480" class="size-full wp-image-27365 c4" /><p id="caption-attachment-27365" class="wp-caption-text">Figure 4: S3 bucket prefix for forensic kernel modules</p></div></li><li>Open each tool name prefix in S3 to find the actual module, such as in the following examples:<ul><li>LiME example: lime-4.14.104-95.84.amzn2.x86_64.ko</li><li>Volatility2 example: 4.14.104-95.84.amzn2.x86_64.zip</li></ul></li></ol><p>Now that the objects have been created, the solution has successfully completed.</p><h2>Incorporate forensic module builds into an EC2 AMI pipeline</h2><p>Each organization has specific requirements for allowing application teams to use various EC2 AMIs, and organizations commonly <a href="https://docs.aws.amazon.com/imagebuilder/latest/userguide/start-build-image-pipeline.html" target="_blank" rel="noopener noreferrer">implement an EC2 image pipeline using tools like EC2 Image Builder</a>. EC2 Image Builder uses <a href="https://docs.aws.amazon.com/imagebuilder/latest/userguide/create-image-recipes.html" target="_blank" rel="noopener noreferrer">recipes</a> to install and configure required components in the AMI before application teams can launch EC2 instances in their environment.</p><p>The EC2 forensic module factory solution we implemented here makes use of an existing EC2 instance AMI. As mentioned, the solution uses an SSM document to create forensic modules. The logic in the SSM document could be incorporated into your EC2 image pipeline to create the forensic modules and store them in an S3 bucket. S3 also allows additional layers of protection such as enforcing <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/default-bucket-encryption.html" target="_blank" rel="noopener noreferrer">default bucket encryption</a> with an <a href="https://docs.aws.amazon.com/mgn/latest/ug/ebs-encryption-kms.html" target="_blank" rel="noopener noreferrer">AWS Key Management Service Customer Managed Key (CMK)</a>, verifying <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html" target="_blank" rel="noopener noreferrer">S3 object integrity</a> with checksum, <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html" target="_blank" rel="noopener noreferrer">S3 Object Lock</a>, and <a href="https://aws.amazon.com/premiumsupport/knowledge-center/secure-s3-resources/" target="_blank" rel="noopener noreferrer">restrictive S3 bucket policies</a>. These protections can help you to ensure that your forensic modules have not been modified and are only accessible by authorized entities.</p><p>It is important to note that incorporating forensic module creation into an EC2 AMI pipeline will build forensic modules for the specific kernel version used in that AMI. You would still need to employ this EC2 forensic module solution to build a specific forensic module version if it is missing from the S3 bucket where you are creating and storing these forensic modules. The need to do this can arise if the EC2 instance is updated after the initial creation of the AMI.</p><h2>Incorporate the solution into existing EC2 incident response automation</h2><p>There are many existing solutions to automate incident response workflow for quarantining and capturing forensic evidence for EC2 instances, but the majority of EC2 incident response automation solutions have a single dependency in common, which is the use of specific forensic modules for the target EC2 instance kernel version. The EC2 forensic module factory solution in this post enables you to be both proactive and reactive when building forensic kernel modules for your EC2 instances.</p><p>You can use the EC2 forensic module factory solution in two different ways:</p><ol><li><strong>Ad-hoc</strong> – In this post, you walked through the solution by running the Step Functions workflow with specific parameters. You can do this to build a repository of kernel modules.</li><li><strong>Automated</strong> – Alternatively, you can incorporate this solution into existing automation by invoking the Step Functions workflow and passing the AMI ID and kernel version. An example could be the following:<ol class="c6"><li>An existing EC2 incident response solution attempts to get the forensic modules to capture the volatile memory from an S3 bucket.</li><li>If the specific kernel version is missing in the S3 bucket, the solution updates the automation to <a href="https://docs.aws.amazon.com/step-functions/latest/apireference/API_StartExecution.html" target="_blank" rel="noopener noreferrer">StartExecution</a> on the <strong>create_ec2_volatile_memory_modules</strong> state machine.</li><li>The Step Functions workflow builds the specific forensic modules.</li><li>After the Step Functions workflow is complete, the EC2 incident response solution restarts its workflow to get the forensic modules to capture the volatile memory on the EC2 instance.</li></ol></li></ol><p>Now that you have the kernel modules, you can both capture the volatile memory by using <a href="https://github.com/504ensicsLabs/LiME#example" target="_blank" rel="noopener noreferrer">LiME</a>, and then conduct analysis on the memory dump by using a <a href="https://github.com/volatilityfoundation/volatility/wiki/Volatility-Usage" target="_blank" rel="noopener noreferrer">Volatility2</a> profile.</p><p><strong>To capture and analyze volatile memory on the target EC2 instance (high-level steps)</strong></p><ol><li>Copy the LiME module from the S3 bucket holding the module repository to the target EC2 instance.</li><li>Capture the volatile memory by using the LiME module.</li><li>Stream the volatile memory dump to a S3 bucket.</li><li>Launch an EC2 forensic workstation instance, with Volatility2 installed.</li><li>Copy the Volatility2 profile from the S3 bucket to the appropriate location.</li><li>Copy the volatile memory dump to the EC2 forensic workstation.</li><li>Run analysis on the volatile memory with Volatility2 by using the specific Volatility2 profile created for the target EC2 instance.</li></ol><h2>Automated self-service AWS solution</h2><p>AWS has also released the <a href="https://docs.aws.amazon.com/solutions/latest/automated-forensics-orchestrator-for-amazon-ec2/welcome.html" target="_blank" rel="noopener noreferrer">Automated Forensics Orchestrator for Amazon EC2</a> solution that you can use to quickly set up and configure a dedicated forensics orchestration automation solution for your security teams. The Automated Forensics Orchestrator for Amazon EC2 allows you to capture and examine the data from EC2 instances and attached <a href="https://aws.amazon.com/ebs/" target="_blank" rel="noopener noreferrer">Amazon Elastic Block Store (Amazon EBS)</a> volumes in your AWS environment. This data is collected as forensic evidence for analysis by the security team.</p><p>The Automated Forensics Orchestrator for Amazon EC2 creates the foundational components to enable the EC2 forensic module factory solution’s <a href="https://docs.aws.amazon.com/solutions/latest/automated-forensics-orchestrator-for-amazon-ec2/solution-components.html#memory-forensics-acquisition-workflow-implementation" target="_blank" rel="noopener noreferrer">memory forensic acquisition workflow</a> and <a href="https://docs.aws.amazon.com/solutions/latest/automated-forensics-orchestrator-for-amazon-ec2/solution-components.html#forensic-investigation-and-reporting-service-1" target="_blank" rel="noopener noreferrer">forensic investigation and reporting service</a>. Both the Automated Forensics Orchestrator for Amazon EC2, and the EC2 forensic module factory, are hosted in different GitHub projects. And you will need to reconcile the expected S3 bucket locations for the associated modules:</p><ul><li>Automated Forensics Orchestrator for Amazon EC2 modules: <a href="https://github.com/aws-solutions/automated-forensic-orchestrator-for-amazon-ec2/blob/main/source/ssm-documents/linux_lime-memory-acquisition.json#L101" target="_blank" rel="noopener noreferrer">S3 bucket location for LiME</a> and <a href="https://github.com/aws-solutions/automated-forensic-orchestrator-for-amazon-ec2/blob/main/source/ssm-documents/amazon-linux-2-volatility-profile.json#L74" target="_blank" rel="noopener noreferrer">S3 bucket location for Volatility2</a></li><li>EC2 forensic module factory modules: <a href="https://github.com/aws-samples/ec2-forensic-module-factory/blob/main/ec2-forensic-module-factory/lib/cdk-module-factory-stack.ts#L266" target="_blank" rel="noopener noreferrer">S3 bucket location for LiME</a> and <a href="https://github.com/aws-samples/ec2-forensic-module-factory/blob/main/ec2-forensic-module-factory/lib/cdk-module-factory-stack.ts#L276" target="_blank" rel="noopener noreferrer">S3 bucket location for Volatility2</a></li></ul><h2>Customize the EC2 forensic module factory solution</h2><p>The SSM document pulls open-source packages to build tools for the specific Linux kernel version. You can update the SSM document to your specific requirements for forensic analysis, including expanding support for other operating systems, versions, and tools.</p><p>You can also update the S3 object naming convention and object tagging, to allow external solutions to reference and copy the appropriate kernel module versions to enable the forensic workflow.</p><h2>Clean up</h2><p>If you deployed the EC2 forensic module factory solution by using the <strong>Launch Stack</strong> button in the AWS Management Console or the CloudFormation template ec2_module_factory_cfn, do the following to clean up:</p><ol><li>In the <a href="https://us-east-1.console.aws.amazon.com/cloudformation/home?region=us-east-1" target="_blank" rel="noopener noreferrer">AWS CloudFormation console</a> for the account and Region where you deployed the solution, choose the <strong>Ec2VolModules</strong> stack.</li><li>Choose the option to <strong>Delete</strong> the stack.</li></ol><p>If you deployed the solution by using the AWS CDK, run the following command.</p><p><code class="c11">cdk destroy</code></p><h2>Conclusion</h2><p>In this blog post, we walked you through the deployment and use of the EC2 forensic module factory solution to use AWS Step Functions, AWS Lambda, AWS Systems Manager, and Amazon EC2 to create specific versions of forensic kernel modules for Amazon Linux EC2 instances.</p><p>The solution provides a framework to create the foundational components required in an EC2 incident response automation solution. You can customize the solution to your needs to fit into an existing EC2 automation, or you can deploy this solution in tandem with the <a href="https://docs.aws.amazon.com/solutions/latest/automated-forensics-orchestrator-for-amazon-ec2/welcome.html" target="_blank" rel="noopener noreferrer">Automated Forensics Orchestrator for Amazon EC2</a>.</p><p>If you have feedback about this post, submit comments in the <strong>Comments</strong> section below. If you have any questions about this post, start a thread on <a href="https://repost.aws/tags/TAbM0xYgFuTsOtxRDmkvrhMA/incident-response" target="_blank" rel="noopener noreferrer">re:Post</a>.</p><p><strong>Want more AWS Security news? Follow us on <a title="Twitter" href="https://twitter.com/AWSsecurityinfo" target="_blank" rel="noopener noreferrer">Twitter</a>.</strong></p></section>
![https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2022/09/23/img1-1-1024x704.png"](https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2022/09/23/img1-1-1024x704.png"){kind=link}
![https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2019/06/05/launch-stack-button.png"](https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2019/06/05/launch-stack-button.png"){kind=link}
![https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2022/09/23/img2-3-1024x309.png"](https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2022/09/23/img2-3-1024x309.png"){kind=link}
![https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2022/09/23/img3-3.png"](https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2022/09/23/img3-3.png"){kind=link}
![https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2022/09/23/img4-1.png"](https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2022/09/23/img4-1.png"){kind=link}