US: +1 888 988-8647UK: +44 0870 777 1500

Sandbox Evidence, Testing, and Remediation Guide - AWS

How Control Testing Works

In summary, a Control Test is the programmatic collection and assessment of data from tested components to satisfy control objectives. One or more Control Tests run in a Test Suite on a scheduled interval. For each Control Test, a population is retrieved and sampled, then evidence is collected and assessed. Test outcomes are available in the Continuous Controls Power BI app. Evidence is located solely inside the customer evidence repository.

One or more Test Cases are executed as part of a logical group, called a Test Suite. Test Suites can be scheduled on manual, daily, weekly, monthly, or quarterly intervals. During the initial delivery, your Test Suites will run daily, usually around noon CST. Test results are accessible in your Continuous Controls Power BI Dashboard (more details below). For each Control Test included in a Test Suite during a Test Run, steps taken include:

  1. Customer Bots collect population and evidence from the customer-managed systems. All data is stored within the customer’s Evidence Repository.

  2. Logical assessments are performed on the evidence to evaluate if the component passes or fails the control objective.

  1. If failed, and a remediation bot exists, Workflow IQ sends an approval request to your team, and if approved, the bot then fixes the problem.

  2. If failed, and a remediation bot exists, Workflow IQ sends an approval request to your team, and if not approved, a manual remediation task is opened for your team.

  3. If failed, and no remediation bot exists, Workflow IQ opens a manual remediation task for your team.

How Bots Work

Continuous Controls uses customer-owned serverless computing and secure key management infrastructure for secure evidence collection and remediation tasks.

Bots

The Continuous Controls platform relies on customer “bots” to perform evidence collection and remediation tasks. Bots are customer-owned functions running in the AWS Lambda or Azure Functions serverless computing service. Each function is an independent unit of deployment, like a microservice, simply code, deployed in the cloud. A bot typically has a single function, such as collecting a particular evidence type, or ameliorating a particular failed condition. A single Control Test with remediation may use up to three different bots to operate, one for subjects, one for evidence, and another for remediation.

Secure Key Management

The Bots require secure API keys to operate. These keys are stored as “secrets” in a customer-owned secure key management service such as AWS Secrets Manager, or Azure Key Vault. When Continuous Controls platform needs evidence, it first calls on this service to retrieve the API key, then it uses this API key to securely call the customer bot. In this way, Neverfail operates in a fully secure, zero-trust architecture. The customer fully controls access to secrets using fine-grained permissions, and can revoke access at any time.

Workflow IQ

Workflow IQ (WIQ) securely connects customer task, approval, and storage systems to the Continuous Controls platform. The customer can use a variety of supported services for approval, task, and storage features, such as JIRA, Slack, and S3 object storage, on a per-team basis. WIQ Tasks, Approval Requests, and Storage usage reporting and team management features are available in the WIQ web UI.

WIQ Approvals

During the initial delivery, WIQ will be used for automated remediation approvals. When a test fails, if an automated remediation bot exists, Continuous Controls will send an approval request to the team’s approval feature, such as a Slack channel. The request will provide details of the failed test and audited component, and enable users to approve or deny an automated remediation.

WIQ Tasks

During the initial delivery, WIQ will also be used for manual remediation tasks. When a test fails, if no automated remediation bot exists, Continuous Controls will send a manual remediation task to the team’s task feature, such as Jira.

WIQ Storage

During the initial delivery, WIQ will be used to store and serve customer data. For example, during test evidence collection, Continuous Controls will call a customer’s collection bot to collect the evidence from the customer’s audited component. Inside the collection bot, WIQ writes the data inside the customer’s evidence repository, which is simply a WIQ team’s configured storage feature on an S3 bucket or Azure blob container. Though zero customer data is stored in the platform, evidence metadata is available in Continuous Controls to enable future requests for customer evidence via WIQ. During the assessment portion of a Control Test, Continuous Controls uses the metadata to identify the location of the evidence, then calls WIQ to securely access it.

The Continuous Controls Power BI App

The Continuous Controls Power BI App can be used to see reports with Control Testing details and outcomes over time. To access the app, please visit https://app.powerbi.com and open the app per the earlier deployment. Reports included:

  • Test Suites

  • Test Runs

  • Control Testing

  • Component Testing

  • Test Cases

  • Sync

Data Synchronization

Continuous Controls events such as Test Suite runs trigger data updates to the Neverfail CC BI database (CCBI DB). Updates are also performed hourly from Neverfail to CCBI DB. The “Sync” report in the Power BI app will show the latest update time.

Power BI imports the data from the CCBI DB into its dataset on a scheduled frequency and on-demand. You might consider such a dataset a point-in-time copy. Because Power BI copies the data, you must refresh the dataset to fetch changes from the CCBI DB. Up to eight scheduled refreshes per day can be configured in PowerBI.

Test Suite Runs

From the Test Suites report, you can launch new Test Suite Runs. When clicked, the Run Test Suite button will notify the Continuous Controls platform to queue up a new Test Suite run. The actual execution of your Test Suite is determined by its place in the queue, and it may not run immediately, but generally should run within five minutes or less. Upon completion, updated data is sent to the CCBI DB, with a new sync time, and the PowerBI app dataset is ready to be refreshed. Currently, this feature allows up to five Test Suite Runs per 24-hour period, resetting the timer at 8am cst. These runs are in addition to the normal scheduled runs, which may be daily, weekly, monthly, or quarterly.

Control Test Walkthrough

As part of the initial delivery, we will walk through a sample AWS Control Test from start to finish to learn how the collection bots interact with customer components to create population and evidence, how that evidence is assessed, and how to locate the evidence files in the customer Evidence Repository. For this Control Test walkthrough, we will use test “Production Relational Database Service (RDS) Databases Have Secure Configuration.” A common use of a DB instance in a VPC is to share data with an application server that is running in an EC2 instance in the same VPC. In this configuration, the RDS DB should not be publicly accessible, which is among the assessment criteria for this Control Test.

Control Test Walkthrough Goals

  • Understand population and evidence retrieval

  • Understand automated assessment logic

  • Locate evidence in the evidence repository, including the Evidence Chain of Custody

Control Test Details

Population Retrieval

The population for this test is the set of RDS DBaaS instances tagged with the “environment”:“nfcc-production” tag. The customer collection bot uses APIs to retrieve the matching instances from AWS RDS. Once the population data is collected, WIQ will store it in the evidence repository as JSON documents.

{
"dbName":"sitest-db-1",
"dbEngine":"oracle-se1",
"dbEngineVersion":"11.2.0.4.v24",
"dbEndpoint":"sitest-db-1.celahfvcgyll.us-east-2.rds.amazonaws.com",
"dbInstanceArn":"arn:aws:rds:us-east-2:876393867294:db:sitest-db-1",
"awsRegionId":"us-east-2",
"awsRegionName":"US East (Ohio)",
"environment":"nfcc-production"
}

Example RDS Population JSON Document

Population Sampling

Test subjects are sampled from a population before a Control Test. The sampling rate is adjustable. The initial delivery defaults to 100% sampling, meaning the entire population is subject to control testing.

Evidence Retrieval

Because this Control Test is assessing several security configuration settings, the required evidence consists of a structured list of those settings for that particular DB. The next step is to retrieve this evidence for each subject, using another collection bot. Again, once the evidence data is collected, WIQ will store it in the evidence repository as JSON documents.

{
"assetId": "sitest-db-1.celahfvcgyll.us-east-2.rds.amazonaws.com",
"assetType": "rdsInstance",
"assetName": "sitest-db-1",
"backupRetentionPeriodDays": 0,
"assetStorageEncrypted": true,
"assetPublicAccess": true,
"networkTcpPort": 1521,
"awsRegionId": "us-east-2"
}

Example DBaaS Instance Security Configuration Evidence JSON Document

Automated Assessment

For each Control Test, a logical assessment is performed on the collected evidence to determine if the required conditions are satisfied (pass) or not (fail). For this assessment, we are checking the DBaaS instance’s public accessibility, backup retention, and storage encryption settings. You will notice that the evidence includes the network TCP port of the DBaaS instance as an informational datapoint, however it is not assessed. If your organization requires certain commonly-exploited ports were banned from use, such as TCP 1433, we would implement this evidence type in a different Control Test, one that evaluates that port for policy violations. For every Control Test, the assessment logic required is described in “Gherkin,” a business-readable, ordinary language parser. The “Gherkin” for this Control Test is shown below:

Scenario: Check if RDS DBs are secured
Given a DBaaS instance as subject
And a security config policy as evidence
When I get security config values
Then value of backupRetentionPeriodDays should be greater than 0
And value of assetStorageEncrypted should be true
And value of assetPublicAccess should be false

Example of Gherkin

Evidence Review

With the assessment completed, the Control Test is now done and the Evidence Chain of Custody is being generated. For each piece of evidence, Continuous Controls automatically synthesizes its collection logs into an Evidence Chain of Custody, a process which takes up to seven minutes to complete. The Evidence Chain of Custody is itself considered another piece of evidence and is also stored in the Evidence Repository. To access the Evidence Repository directly, use the AWS S3 portal to navigate to the S3 bucket, as configured in the WIQ storage team feature. From here, you will see a list of objects, each of these being either a piece of evidence or an Evidence Chain of Custody. Feel free to select, then Open any of them. Generally, the ones that are of a larger size will be the Evidence Chain of Custody records.

Manual Remediation Walkthrough

We will now walk through a sample manual remediation task to learn how to manually remediate a failed test condition. For this manual remediation sample review, we will continue to use the test from the last section, “Production Relational Database Service (RDS) Databases Have Secure Configuration.” Manual remediation tasks are the default outcome for failed tests where no remediation bot exists for that particular failed condition. In this walkthrough, because we do have a remediation bot, we will have to reject the automated remediation WIQ Approval to get us to a manual remediation task.

Manual Remediation Walkthrough Goals

  • Understand how changes to the infrastructure alter control test outcomes

  • Understand how to perform a manual remediation WIQ Task

Preparing for Manual Remediation

In order to exhibit a manual remediation process using WIQ, we will need to set failing conditions for the assessment, by setting public accessibility to “true” for our RDS DBaaS instance.

  1. From the Amazon RDS console, navigate to the database list by clicking Databases on the left.

  2. Click on the database that has a label similar to “<customer-name>-db-1.”

  3. Click the Modify button on the upper-right of the window.

  1. Scroll down to the “Connectivity” section, and look for the “Public Access” settings, then set the option to Publicly accessible.
    (You may need to expand a section called “Additional connectivity configuration” to access this setting).

  1. Scroll down further and click Continue.








  2. On the next screen, set Scheduling of modifications to Immediately, then click Modify DB instance.

  1. Because AWS RDS takes some time to enact the changes, please wait 1-2 minutes and verify that the setting has changed by refreshing the database view until you see Public accessibility setting set to “Yes.”



  1. Now trigger another Test Run to see the test in a failed assessment state:

    • Navigate to the Test Suite view

    • Run the AWS Compliance Test Suite by clicking on the Run Test Suite button

    This will notify the Continuous Controls platform to queue up a new Test Suite run.





  2. Because a remediation bot exists to fix this condition, a WIQ Approval Request will open in your configured WIQ Approval system. It could take up to five minutes to see the Approval request. Reject this request so that WIQ opens a manual remediation task instead.

  1. Now that the test is complete, and CC BI DB updated, refresh your dataset in your Power BI app to pull the latest copy of data. Please visit Appendix A at the end of this document to review your dataset refresh methods.

  2. Check the Sync report to see when the last data sync occurred between the testing platform and the CC BI DB. It should reflect the most recent run time. If it doesn’t, refresh the dataset again.

  3. Now trigger another Test Run to see the test in a failed assessment state:
    • Navigate to the Test Suite view

    • Run the AWS Compliance Test Suite by clicking on the Run Test Suite button



  1. Now inside the app, click on the Control Testing tab, and you should see a failed test. If you do not see this failed test, please try clicking the Refresh button above the dashboard.

  1. Click on the link icon under the Evidence column for the failed test to see the JSON document evidence. As you can see, assetPublicAccess is set to true, which fails the assessment.

    {
    "assetId": "sitest-db-1.celahfvcgyll.us-east-2.rds.amazonaws.com",
    "assetType": "rdsInstance",
    "assetName": "sitest-db-1",
    "backupRetentionPeriodDays": 0,
    "assetStorageEncrypted": true,
    "assetPublicAccess": true,
    "networkTcpPort": 1521,
    "awsRegionId": "us-east-2",
    "awsRegionName": "US East (Ohio)"
    }

    Example Failing DBaaS Instance Security Configuration Evidence JSON Document

Manual Remediation Using the WIQ Task

At this point you should now have a new remediation task in your configured WIQ task system. To remediate this test condition, you need to set that RDS DBaaS instance to not be publicly accessible, then close the task.

  1. Visit your configured Task system to see your new remediation task.

  2. Repeat steps 1-4 from the previous section (above), but now set Public Access to “Not publicly accessible.”

  3. Scroll down and click Continue.




  4. On the next screen, set Scheduling of modifications to Immediately and click Modify DB instance.

  1. Because AWS RDS takes some time to enact the changes, please wait 1-2 minutes and then verify this change by refreshing the database view until the Public accessibility setting set to “No”.

  2. Complete the WIQ Task in your task system.

  3. Now trigger another Test Run to see the test in a passed assessment state. Go to the Power BI app (http://app.powerbi.com/), navigate to the Test Suite view, then run the AWS Compliance Test Suite.

  4. Wait five minutes for the test to run and update the CC BI DB. Refresh your dataset in your Power BI app to pull the latest copy of data. You can refresh as often as needed.

  5. Now inside the app, visit Control Testing, and you should see the passed test. If you do not see this passed test, please try clicking the Refresh button above the dashboard.

  6. On the Control Testing report, feel free to click on the link icon under the Evidence column for the passed test to see the JSON document evidence. Now the assetPublicAccess is set to false, which passes the assessment.

    {
    "assetId": "sitest-db-1.celahfvcgyll.us-east-2.rds.amazonaws.com",
    "assetType": "rdsInstance",
    "assetName": "sitest-db-1",
    "backupRetentionPeriodDays": 0,
    "assetStorageEncrypted": true,
    "assetPublicAccess": false,
    "networkTcpPort": 1521,
    "awsRegionId": "us-east-2",
    "awsRegionName": "US East (Ohio)"
    }

    Example Passing DBaaS Instance Security Configuration Evidence JSON Document

Automated Remediation Walkthrough

We will now walk through another sample AWS Control Test to demonstrate how automated remediation works. For this automated remediation sample, we will use test: “Public Access Blocked on S3 Cloud Storage.”

Automated Remediation Walkthrough Goals

  • Break the conditions of the assessment on a Control Test to trigger an automated remediation

  • Understand how automated remediation operates through WIQ

Control Test Details

Preparing for Automated Remediation

In order to exhibit an automated remediation process using WIQ, we need to enable public access to a production S3 object storage bucket to cause the next assessment to fail.

  1. From the AWS console, select Storage -> S3 (or search for “S3”) to access the Amazon S3 console.

  2. Review the bucket list and click on the one with label “CUSTOMER_NAME-neverfail-s3-production-object-storage-1.”
    (This is an S3 bucket with a special tag applied of “environment”:”nfcc-production,” which simulates tag application in the real-world.)

  3. On the bucket detail view, click the Permissions tab to view the access settings.

  4. Under the Block public access section, click the Edit button on the far right-hand side.

  5. Uncheck Block all public access checkbox, then click Save.

  1. If you get a confirmation window, follow the instructions.

  1. Now we trigger another Test Run so we can see the test in a failed assessment state. Go to the Power BI app (http://app.powerbi.com/), navigate to the Test Suite view, then run the AWS Compliance Test Suite.

  2. Because a remediation bot exists to fix this condition, a WIQ approval request will be sent to your configured WIQ Approval system. It could take up to five minutes to see the Approval request. Open your configured task system and Approve this request so that the remediation bot runs.

  1. When prompted, input time spent in minutes. The remediation bot will be called and the configuration ameliorated.

  2. Now that the test is complete, and CC BI DB updated, refresh your dataset in your Power BI app to pull the latest copy of data. Please visit Appendix A at the end of this document to review your dataset refresh methods.

  3. Check the Sync report to see when the last data sync occurred between Neverfail and the CC BI DB. It should reflect the most recent Test Suite run time. If it does not, refresh the dataset again.

  1. Now trigger another Test Run to see the test in a passed assessment state. Go to the Power BI app (http://app.powerbi.com/), navigate to the Test Suite view, then run the AWS Compliance Test Suite.

  2. Wait five minutes for the test to run and update the CC BI DB.
    Refresh your dataset in your Power BI app to pull the latest copy of data. You can refresh as often as needed.

  3. Now inside the app, visit Control Testing, and you should see the passed test. If you do not see this passed test, please try clicking the Refresh button above the dashboard.

  4. On the Control Testing report, feel free to click on the link icon under the Evidence column for the passed test to see the JSON document evidence. The block “publicAcls” are set to false, which fails the assessment.

Evidence Review Walkthrough

Before proceeding, we recommend you have a JSON document formatter web browser extension to ensure the data files are legible. For Chrome, some options include JSONView, JSONFormatter, or PrettyJSON (though we do not recommend any particular extension).

  1. From the AWS console, select Storage -> S3 (or search for “S3”) to access the Amazon S3 console.

  2. Review the bucket list and locate one with the label “CUSTOMER_NAME-neverfail-s3-evidence-repository-1.”
    (This is your Evidence Repository.)

  3. Click on the bucket link, then click again on the folder within it.

  4. From here, you will see a list of objects, each of these being either a piece of evidence or an Evidence Chain of Custody. Feel free to select, then Open any of them. Generally, the ones that are of a larger size will be the Evidence Chain of Custody records.

Congratulations! You've completed the Sandbox Tutorial.

Appendix A

Data Refreshes in Power BI

You can use on-demand data refreshes to update the data in the Continuous Controls Power BI app.

On-demand Data Refresh

  1. In the app, click the pencil icon on the upper-right to edit settings.

  1. Click the Datasets label then click the Refresh icon, per screenshot below.

Still Not Updating?

Please note that even after a dataset refresh, your app’s report views may show outdated data. If this happens, please hit the refresh button on the report view, per screenshot below.

Reviewing Latest Sync Time

On the Sync report in the app, you will find the timestamp of the latest data update event from theContinuous Controls platform to the CC BI server, from where your PowerBI app retrieves its copy of data.

TOP