US: +1 888 988-8647

Sandbox Evidence, Testing, and Remediation Guide - Azure

How Control Testing Works

In summary, a Control Test is the programmatic collection and assessment of data from tested components to satisfy control objectives. One or more Control Tests run in a Test Suite on a scheduled interval. For each Control Test, a population is retrieved and sampled, then evidence is collected and assessed. Test outcomes are available in the Continuous Controls Power BI app. Evidence is located solely inside the customer evidence repository.

One or more Test Cases are executed as part of a logical group, called a Test Suite. Test Suites can be scheduled on manual, daily, weekly, monthly, or quarterly intervals. During this deployment, your Test Suites will run daily, usually around noon CST. Test results are accessible in your Power BI Dashboard (more details below). For each Control Test included in a Test Suite during a Test Run, steps taken include:

  1. Customer Bots collect population and evidence from the customer-managed systems. All data is stored within the customer’s Evidence Repository.

  2. Logical assessments are performed on the evidence to evaluate if the component passes or fails the control objective.

  1. If failed, and a remediation bot exists, Workflow IQ sends an approval request to your team, and if approved, the bot then fixes the problem.

  2. If failed, and a remediation bot exists, Workflow IQ sends an approval request to your team, and if not approved, a manual remediation task is opened for your team.

  3. If failed, and no remediation bot exists, Workflow IQ opens a manual remediation task for your team.

How Bots Work

Continuous Controls uses customer-owned serverless computing and secure key management infrastructure for secure evidence collection and remediation tasks.

Bots

The Continuous Controls platform relies on customer “bots” to perform evidence collection and remediation tasks. Bots are customer-owned functions running in the AWS Lambda or Azure Functions serverless computing service. Each function is an independent unit of deployment, like a microservice, simply code, deployed in the cloud. A bot typically has a single function, such as collecting a particular evidence type, or ameliorating a particular failed condition. A single Control Test with remediation may use up to three different bots to operate, one for subjects, one for evidence, and another for remediation.

Secure Key Management

The Bots require secure API keys to operate. These keys are stored as “secrets” in a customer-owned secure key management service such as AWS Secrets Manager, or Azure Key Vault. When Continuous Controls platform needs evidence, it first calls on this service to retrieve the API key, then it uses this API key to securely call the customer bot. In this way, Neverfail operates in a fully secure, zero-trust architecture. The customer fully controls access to secrets using fine-grained permissions, and can revoke access at any time.

Workflow IQ

Workflow IQ (WIQ) securely connects customer task, approval, and storage systems to the Continuous Controls platform. The customer can use a variety of supported services for approval, task, and storage features, such as JIRA, Slack, and S3 object storage, on a per-team basis. WIQ Tasks, Approval Requests, and Storage usage reporting and team management features are available in the WIQ web UI.

WIQ Approvals

During the deployment, WIQ will be used for automated remediation approvals. When a test fails, if an automated remediation bot exists, Continuous Controls will send an approval request to the team’s approval feature, such as a Slack channel. The request will provide details of the failed test and audited component, and enable users to approve or deny an automated remediation.

WIQ Tasks

During the deployment, WIQ will also be used for manual remediation tasks. When a test fails, if no automated remediation bot exists, Continuous Controls will send a manual remediation task to the team’s task feature, such as Jira.

WIQ Storage

During the deployment, WIQ will be used to store and serve customer data. For example, during test evidence collection, Continuous Controls will call a customer’s collection bot to collect the evidence from the customer’s audited component. Inside the collection bot, WIQ writes the data inside the customer’s evidence repository, which is simply a WIQ team’s configured storage feature on an S3 bucket or Azure blob container. Though zero customer data is stored in the platform, evidence metadata is available in Continuous Controls to enable future requests for customer evidence via WIQ. During the assessment portion of a Control Test, Continuous Controls uses the metadata to identify the location of the evidence, then calls WIQ to securely access it.

The Continuous Controls Sandbox Deployment Power BI App

The Sandbox Deployment Power BI App can be used to see reports with Control Testing details and outcomes over time. To access the app, please visit https://app.powerbi.com and open the app per the earlier deployment. Reports included:

  • Test Suites

  • Test Runs

  • Control Testing

  • Component Testing

  • Test Cases

  • Sync

Data Synchronization

Continuous Controls events such as Test Suite runs trigger data updates to the Neverfail CC BI database (CCBI DB). Updates are also performed hourly from Neverfail to CCBI DB. The “Sync” report in the Power BI app will show the latest update time.

Power BI imports the data from the CCBI DB into its dataset on a scheduled frequency and on-demand. You might consider such a dataset a point-in-time copy. Because Power BI copies the data, you must refresh the dataset to fetch changes from the CCBI DB. Up to eight scheduled refreshes per day can be configured in PowerBI.

Test Suite Runs

From the Test Suites report, you can launch new Test Suite Runs. When clicked, the Run Test Suite button will notify the Continuous Controls platform to queue up a new Test Suite run. The actual execution of your Test Suite is determined by its place in the queue, and it may not run immediately, but generally should run within five minutes or less. Upon completion, updated data is sent to the CCBI DB, with a new sync time, and the PowerBI app dataset is ready to be refreshed. Currently, this feature allows up to five Test Suite Runs per 24-hour period, resetting the timer at 8am cst. These runs are in addition to the normal scheduled runs, which may be daily, weekly, monthly, or quarterly.

Control Test Walkthrough

As part of the deployment, we will walk through a sample Azure Control Test from start to finish to learn how the collection bots interact with customer components to create population and evidence, how that evidence is assessed, and how to locate the evidence files in the customer Evidence Repository. For this Control Test walkthrough, we will use test “Private and Secure Azure Blob Storage.” For secure use cases, Microsoft recommends that blob containers have HTTPS-only transport, server-side encryption, and disallowed public access, which are the assessment criteria for this Control Test.

Control Test Walkthrough Goals

  • Understand population and evidence retrieval

  • Understand automated assessment logic

  • Locate evidence in the evidence repository, including the Evidence Chain of Custody

Control Test Details

Population Retrieval

The population for this test is the set of Azure Storage Blob Containers contained in production resource groups (resource groups with labels including the word “production”). Note that in the paid service, filtering of this component can be achieved using labels or tags. The customer collection bot uses APIs to retrieve the matching instances from Azure Storage Services. Once the population data is collected, WIQ will store it in the evidence repository as JSON documents.

{
"storageAccountName":"july9azuproductionsa1",
"resourceGroup":"july9azu-production-rg-1",
"subscriptionId":"66e8cbf4-3e03-4b9e-9548-9b620f3d37b6",
"containerName":"july9azu-neverfail-azure-production-object-storage-1",
"url":"https://july9azuproductionsa1.blob.core.windows.net/july9azu-neverfail-azure-production-object-storage-1"
}

Example Azure Storage Blob Container Population JSON Document

Population Sampling

Test subjects are sampled from a population before a Control Test. The sampling rate is adjustable. The Sandbox Deployment defaults to 100% sampling, meaning the entire population is subject to control testing.

Evidence Retrieval

In this Control Test, the required evidence consists of three configuration checks including HTTPS-only transport setting, server-side encryption setting, and public access setting. The next step is to retrieve these “Azure Storage Security Policy” configurations for each subject, using another collection bot. Again, once the evidence data is collected, WIQ will store it in the evidence repository.

{
"httpsOnlyTransport": true,
"isServerSideEncryptionEnabled": true,
"blockPublicAcls": true,
"containerName": "july9azu-neverfail-azure-production-object-storage-1",
"storageAccountName": "july9azuproductionsa1"
}

Example Azure Storage Security Policy Evidence JSON Document

Automated Assessment

For each Control Test, a logical assessment is performed on the collected evidence to determine if the required conditions are satisfied (pass) or not (fail). For this assessment, if each of those settings in the Azure Storage Security Policy configuration are true, this Control Test should pass. The logic required is described in “Gherkin,” a business-readable, ordinary language parser. The “Gherkin” for this Control Test is described below:

Scenario: Check if azure storage accounts from a specific subscription are secured
Given a storage container as subject
And an azure storage security policy as evidence
When I get values of httpsOnlyTransport, isServerSideEncryptionEnabled and blockPublicAcls
Then values of httpsOnlyTransport, isServerSideEncryptionEnabled and blockPublicAcls should be true

Example of Gherkin

Evidence Review

With the assessment completed, the Control Test is now done and the Evidence Chain of Custody is being generated. For each piece of evidence, Continuous Controls automatically synthesizes its collection logs into an Evidence Chain of Custody, a process which takes up to seven minutes to complete. The Evidence Chain of Custody is itself considered another piece of evidence and is also stored in the Evidence Repository. To access the Evidence Repository directly, use the Azure Portal’s Storage Explorer to navigate to the blob container as configured in the WIQ storage team feature. From here, you will see a list of objects, each of these being either a piece of evidence or an Evidence Chain of Custody. Feel free to select, then Open any of them. Generally, the ones that are of a larger size will be the Evidence Chain of Custody records.

Manual Remediation Walkthrough

We will now walk through a sample manual remediation task to learn how to manually remediate a failed test condition. For this manual remediation sample review, we will continue to use the test from the last section, “Private and Secure Azure Blob Storage.” Manual remediation tasks are the default outcome for failed tests where no remediation bot exists for that particular failed condition. In this walkthrough, because we do have a remediation bot, we will have to reject the automated remediation WIQ Approval to get us to a manual remediation task.

Please note, for the Azure Compliance Test Suite to function, the AD server VM MUST be in a running state.
The AD server VM name ends with “-dc-1.”

Manual Remediation Walkthrough Goals

  • Understand how changes to the infrastructure alter control test outcomes

  • Understand how to perform a manual remediation WIQ Task

Preparing for Manual Remediation

In order to exhibit a manual remediation process using WIQ, we will change two container security settings to cause the next assessment to fail.

  1. Login to the Azure portal at https://portal.azure.com.

  2. Click on the Storage accounts icon from the Azure services list.

  1. On the next screen, click on the account which is labeled like productionsa1” (with tag NFCC-POC:true).

  1. On the next screen, click on the Configuration menu item under Settings.

  1. Set Secure transfer required to Disabled.

  2. Set Allow Blob public access to Enabled.

  1. Click Save above the settings list.

  2. Now we trigger another Test Run so we can see the test in a failed assessment state. Go to the Power BI app (http://app.powerbi.com/), navigate to the Test Suite view, then run the Azure Compliance Test Suite. Remember, the Run Test Suite button will notify the Continuous Controls platform to queue up a new Test Suite run. The actual execution of your Test Suite is determined by its place in the queue, and it may not run immediately, but generally should run within five minutes or less.

  3. Because a remediation bot exists to fix this condition, a WIQ approval will be opened in your configured WIQ Approval system. It could take up to five minutes to see the Approval request. We need you to Reject this request so that WIQ opens a manual remediation task instead. We will cover WIQ Approvals in more detail in the next section, Automated Remediation Walkthrough.

  1. Now that the test is complete, and the CC BI DB updated, you need to refresh your dataset in your Power BI app to pull the latest copy of data. Please visit Appendix A at the end of this document to review your dataset refresh methods.

  2. Check the Sync report to see when the last data sync occurred between Neverfail and the CC BI DB. It should reflect the most recent Test Suite run time. If it does not, refresh the dataset again.

  1. Now inside the Power BI app, visit Control Testing, and you should see a failed test. If you do not see this failed test, please try clicking the Refresh button above the dashboard.

  1. Click on the link icon under the Evidence column for the failed test to see the JSON document evidence. As you can see, assetPublicAccess is set to true, which fails the assessment.

    {
    "httpsOnlyTransport": false,
    "isServerSideEncryptionEnabled": true,
    "blockPublicAcls": true,
    "containerName": "july9azu-neverfail-azure-production-object-storage-1",
    "storageAccountName": "july9azuproductionsa1"
    }

    Example Failing Azure Storage Security Policy Evidence JSON Document

Manual Remediation Using the WIQ Task

At this point you should now have a new remediation task in your configured WIQ task system. To remediate this test condition, you should undo the settings changes made in the last section. Once done, close as complete the remediation task in your configured team task system.

  1. Login to the Azure portal at https://portal.azure.com.

  2. Click on the Storage accounts icon from the Azure services list.

  3. On the next screen, click on the account which is labeled like <customer_name>productionsa1.”

  4. On the next screen, click on the Configuration menu item under Settings.

  1. Set Secure transfer required to Enabled.

  2. Set Allow Blob public access to Disabled.

  1. Click Save above the settings list.

  2. Complete the WIQ Task in your task system.

  3. Now we trigger another Test Run so we can see the test in a passed assessment state. Go to the Power BI app, navigate to the Test Suite view, then run the Azure Compliance Test Suite.

  4. Wait five minutes for the test to run and update the CC BI DB. Refresh your dataset in your Power BI app to pull the latest copy of data. You can refresh as often as needed.

  5. Now inside the app, visit Control Testing, and you should see a passed test. If you do not see this passed test, please try clicking the Refresh button above the dashboard.

  6. On the Control Testing report, feel free to click on the link icon under the Evidence column for the passed test to see the JSON document evidence.

Automated Remediation Walkthrough

We will now walk through another sample Azure Control Test to demonstrate how automated remediation works. For this automated remediation sample, we will use test: “Private and Secure Azure Blob Storage.”

Please note, for the Azure Compliance Test Suite to function, the AD server VM MUST be in a running state.
The AD server VM name ends with “-dc-1.”

Automated Remediation Walkthrough Goals

  • Break the conditions of the assessment on a Control Test to trigger an automated remediation

  • Understand how automated remediation via WIQ operates

Control Test Details

Preparing for Automated Remediation

In order to exhibit a manual remediation process using WIQ, we will again change two container security settings to cause the next assessment to fail. Note that though this appears similar to the last section, it is slightly different because we will break it, run the test without refreshing the Power BI app, approve the remediation bot to fix it, then finally refresh the Power BI app to see both the recent failed test and the following passed test.

  1. Login to the Azure portal at https://portal.azure.com.

  2. Click on the Storage accounts icon from the Azure services list.

  3. On the next screen, click on the account which is labeled like <customer_name>productionsa1” (with tag NFCC-POC:true).

  4. On the next screen, click on the Configuration menu item under Settings.

  1. Set Secure transfer required to Disabled.

  2. Set Allow Blob public access to Enabled.

  1. Click Save above the settings list.

  2. Now we trigger another Test Run so that later, we can see a “before and after automated remediation” set of tests. Go to the Power BI app (http://app.powerbi.com/), navigate to the Test Suite view, then run the Azure Compliance Test Suite.

  3. Because a remediation bot exists to fix this condition, a WIQ approval will be opened in your configured WIQ Approval system. It could take up to five minutes to see the Approval request. We need you to Approve this request so that WIQ opens a manual remediation task instead.

  1. When prompted, input time spent in minutes. The remediation bot will be called and the configuration ameliorated.

  2. Now that the test is complete, and the CC BI DB updated, you need to refresh your dataset in your Power BI app to pull the latest copy of data. Please visit Appendix A at the end of this document to review your dataset refresh methods.

  3. Check the Sync report to see when the last data sync occurred between Neverfail and the CC BI DB. It should reflect the most recent Test Suite run time. If it does not, refresh the dataset again.

  4. Now inside the app, visit Control Testing, and you should see the passed test. If you do not see this passed test, please try clicking the Refresh button above the dashboard.

Evidence Review Walkthrough

Before proceeding, we recommend you have a JSON document formatter web browser extension to ensure the data files are legible. For Chrome, some options include JSONView, JSONFormatter, or PrettyJSON (though we do not recommend any particular extension).

  1. Login to the Azure portal at https://portal.azure.com.

  2. Click on the Storage accounts icon from the Azure services list.

  3. On the next screen, click on the account which is labeled like evidencesa1” (with tag NFCC-POC:true).

  1. On the next screen, click on Storage Explorer (preview) to access the object browser.

  1. In the Explorer panel, expand BLOB CONTAINERS, then select the container.

  1. If you want to see the contents of a blob, select any of them, then click Open, then on the pop-up window, click Click here to begin download. Generally, the ones that are of a larger size will be the Evidence Chain of Custody records.

Congratulations! You've completed the Primer.

Appendix A

Data Refreshes in Power BI

You can use on-demand data refreshes to update the data in the Continuous Controls Power BI app.

On-demand Data Refresh

  1. In the app, click the pencil icon on the upper-right to edit settings.

  1. Click the Datasets label then click the Refresh icon, per screenshot below.

Still Not Updating?

Please note that even after a dataset refresh, your app’s report views may show outdated data. If this happens, please hit the refresh button on the report view, per screenshot below.

Reviewing Latest Sync Time

On the Sync report in the app, you will find the timestamp of the latest data update event from the Continuous Controls platform to the CC BI server, from where your PowerBI app retrieves its copy of data.

TOP