URL Retrospective Verdict and URL Remediation

Cloud-only

☁️

Cloud-only

URL Retrospective Verdict and URL Remediation in AsyncOS 14.2 is currently a feature only released Cisco Secure Email Cloud Gateway customers (formerly Cisco Cloud Email Security [CES]).

✏️

Note

URL Retrospective Verdict is available for all Gateway (On-premises, Virtual, and Cloud) as of AsyncOS 15.0.

The URLs are filtered based on the URL reputation and category provided by cloud-based Talos Intelligence Services. The URL reputation can change as new information emerges. A URL may not be initially evaluated as malicious, and the message may therefore be released to the recipient. But later, these URL reputations can turn malicious anytime, even after it has reached the user's mailbox. Talos Intelligence Services monitors the URL verdicts in a sandbox server. The email cloud gateway polls for the retrospective verdict update of the URLs from Talos every two minutes for a period of 168 hours (7 days). Talos sends the retrospective verdict update to the email cloud gateway if any URL reputation changes to' Malicious.' The email cloud gateway sends alerts on the retrospective verdict update so that necessary action can be taken.

Cisco Secure Email Cloud Gateway handles the URL Retrospective verdicts generated within seven days of the URLs sent for analysis. The cloud gateway does not perform the configured policy action for verdicts received after seven days.

Additionally, you can configure the Mailbox Auto Remediation service to remediate the message with a malicious URL from the user's mailbox. For example, you can configure your email cloud gateway to delete the message from the recipient's mailbox when the reputation of the URL changes to malicious. The configured policy action is applied only to the delivered messages.

✏️

Note

The URL Retrospective Verdict and Remediation feature is available for incoming mails only.

The URL Retrospective Verdict traffic from Secure Email Cloud Gateway cannot be decrypted. Only pass-through proxy mode is supported. However, the polling response data can be decrypted.

All emails with the same subject line are remediated if one of them contains a malicious URL.

Getting Started

Once the Cloud Gateway has been upgraded to 14.2.0-468 or newer, Security Services > URL Filtering will look like this:

If you have access to the CLI for your Cloud Gateway, you can see the URL Retrospection within URL Filtering, performed with the following steps:

esa1.hcxxyy-zz.iphmx.com> urlretroservice enable

URL Retro Service is enabled.

esa1.hcxxyy-zz.iphmx.com> websecurityconfig

URL Filtering is enabled.
No URL list used.
Web Interaction Tracking is enabled.
URL Retrospective service based Mail Auto Remediation is disabled.
URL Retrospective service status - Unavailable

Disable URL Filtering? [N]>

Do you wish to disable Web Interaction Tracking? [N]>

Do you wish to add URLs to the allowed list using a URL list? [N]>


Enable URL Retrospective service based Mail Auto Remediation to configure remediation actions.

Do you wish to enable Mailbox Auto Remediation action? [N]> y

URL Retrospective service based Mail Auto Remediation is enabled.

Please select a Mailbox Auto Remediation action:
1. Delete
2. Forward and Delete
3. Forward
[1]>

esa1.hcxxyy-zz.iphmx.com> commit

Please enter some comments describing your changes:
[]>

Do you want to save the current configuration for rollback? [Y]>

Changes committed: Tue Mar 29 19:43:48 2022 EDT

The UI will then update to show you:

Once the service has connected, the status will update and the UI will reflect:

No change in service status?

You may need to bounce the 'urlretroservice' via the CLI by running:

  1. urlretroservice disable
  2. commit
  3. urlretroservice enable
  4. commit

You can see the change and success in the ecs log:

esa1.hcxxyy-zz.iphmx.com> tail ecs
Mon Jun 26 13:06:28 2023 Debug: ECS: Starting Retro MAR poller
Mon Jun 26 13:06:28 2023 Debug: ECS: Remediation service notified.
Mon Jun 26 13:06:28 2023 Info: ECS: Starting cloud service polling
Mon Jun 26 13:06:28 2023 Debug: ECS: Initiating remediation
Mon Jun 26 13:06:28 2023 Info: ECS: No data pending for remediation.
Mon Jun 26 13:06:28 2023 Debug: ECS: Start polling
Mon Jun 26 13:06:28 2023 Debug: ECS: Querying URL 'https://produs-retro-api.uce.cmd.cisco.com/v1/retro/device/423AA9781B6766ACXXYY-25CC6C241234' with parameters: {'page_size': 10, 'since': '2023-06-26T12:06:28Z'}
Mon Jun 26 13:06:28 2023 Debug: ECS: Starting Pruning Thread
Mon Jun 26 13:06:28 2023 Info: ECS: Starting URL retrospective polling
Mon Jun 26 13:06:29 2023 Info: ECS: No retrospections found.
Mon Jun 26 13:06:29 2023 Trace: ECS: Cloud retro response: {}
Mon Jun 26 13:06:29 2023 Debug: ECS: Finish polling

Testing URL Retrospective service

Please be sure that you are tailing the following logs via CLI:

  1. ecs (set to 'Trace' level to see detailed logs)
  2. mail_logs
  3. url_rep_client

Looking at the ecs logs:

esa1.hcxxyy-zz.iphmx.com> tail ecs
Tue Mar 29 20:20:24 2022 Debug: ECS: Start polling
Tue Mar 29 20:20:24 2022 Debug: ECS: Querying URL 'https://betaus-retro-api.uce.cmd.cisco.com/v1/retro/device/420D4F36AAEBC0093B4F-B9E72189A021' with parameters: {'since': '2022-03-29T23:20:24Z', 'page_size': 20}
Tue Mar 29 20:20:27 2022 Trace: ECS: URL https://betaus-retro-api.uce.cmd.cisco.com/v1/retro/device/420D4F36AAEBC0093B4F-B9E72189A021?since=2022-03-29T23%3A20%3A24Z&page_size=20, Status 403, Response {"message":"Token Expired"}
Tue Mar 29 20:20:27 2022 Warning: ECS: Cloud query failed. 'Token Expired.'
Tue Mar 29 20:20:27 2022 Debug: ECS: Connecting to register api url https://beta-register-api.uce.cmd.cisco.com/v1/register/device/420D4F36AAEBC0093B4F-B9E72189A021 with payload {'org_id': 'cisco', 'model': 'C300V', 'version': '14.2.0-468', 'type': 'esa-ces'}
Tue Mar 29 20:20:33 2022 Trace: ECS: URL https://beta-register-api.uce.cmd.cisco.com/v1/register/device/420D4F36AAEBC0093B4F-B9E72189A021, Status 201
Tue Mar 29 20:20:33 2022 Info: ECS: Device registration is successful
Tue Mar 29 20:20:33 2022 Debug: ECS: Successfully updated region uri and token in database
Tue Mar 29 20:20:33 2022 Debug: ECS: Updated device registration status
Tue Mar 29 20:20:33 2022 Info: ECS: Token Expired/Invalid. Retrying.
Tue Mar 29 20:20:34 2022 Trace: ECS: Cloud retro response: {'retros': [{'observable': {'observable_type': 'url', 'value': 'http://phish.cua.com/test_url_2022032402rks'}, 'context_tags': [{'taxonomy_id': 4, 'taxonomy_entry_id': 3, 'tag_type_id': 1}], 'email_id': 'd5feb7d4-fe00-4568-9bb7-769a5bc7886e', 'timestamp': '2022-03-30T00:17:13Z', 'org_id': 'cisco', 'verdict': {'taxonomy_id': 4, 'taxonomy_entry_id': 3, 'tag_type_id': 1}, 'taxonomy_map_version': 1}, {'observable': {'observable_type': 'url', 'value': 'http://phish.cua.com/test_url_2022032402rks'}, 'context_tags': [{'taxonomy_id': 4, 'taxonomy_entry_id': 3, 'tag_type_id': 1}], 'email_id': 'c5fb612c-e1b1-42aa-aaa2-643009c803a1', 'timestamp': '2022-03-30T00:19:24Z', 'org_id': 'cisco', 'verdict': {'taxonomy_id': 4, 'taxonomy_entry_id': 3, 'tag_type_id': 1}, 'taxonomy_map_version': 1}, {'observable': {'observable_type': 'url', 'value': 'http://phish.cua.com/test_url_2022032901_rks'}, 'context_tags': [{'taxonomy_id': 4, 'taxonomy_entry_id': 3, 'tag_type_id': 1}], 'email_id': '2ed41b55-bdec-49fd-abfa-45a876fc10d6', 'timestamp': '2022-03-30T00:20:25Z', 'org_id': 'cisco', 'verdict': {'taxonomy_id': 4, 'taxonomy_entry_id': 3, 'tag_type_id': 1}, 'taxonomy_map_version': 1}], 'more': False}
Tue Mar 29 20:20:34 2022 Debug: ECS: Finish polling
Tue Mar 29 20:20:34 2022 Info: URL verdict update received from retrospective service.
Corresponding message ids & URLs are as follows:

<20220330001930.077554@ip-172-31-43-120.us-east-2.compute.internal> : BLOCKEDphish[.]cua[.]com/test_url_2022032901_rksBLOCKED
Tue Mar 29 20:20:34 2022 Debug: ECS: Notifying Remediation Service from Polling Service
Tue Mar 29 20:20:34 2022 Debug: ECS: ECS Remediation service notified
Tue Mar 29 20:20:34 2022 Debug: ECS: Initiating remediation
Tue Mar 29 20:20:34 2022 Info: ECS: No pending data for remediation.

Looking at the mail_logs:

esa1.hcxxyy-zz.iphmx.com> tail mail_logs
Tue Mar 29 20:20:10 2022 Info: New SMTP ICID 535151 interface Data 1 (139.138.39.175) address 3.136.210.164 reverse dns host ec2-3-136-210-164.us-east-2.compute.amazonaws.com verified yes
Tue Mar 29 20:20:10 2022 Info: ICID 535151 ACCEPT SG SUSPECTLIST match sbrs[none] SBRS None country United States
Tue Mar 29 20:20:10 2022 Info: Start MID 63280 ICID 535151
Tue Mar 29 20:20:10 2022 Info: MID 63280 ICID 535151 From: <[email protected]>
Tue Mar 29 20:20:10 2022 Info: MID 63280 SDR: Domains for which SDR is requested: reverse DNS host: ec2-3-136-210-164.us-east-2.compute.amazonaws.com, helo: ip-172-31-43-120.us-east-2.compute.internal, env-from: nowhere.com, header-from: Not Present, reply-to: Not Present
Tue Mar 29 20:20:10 2022 Info: MID 63280 SDR: Consolidated Sender Threat Level: Neutral, Threat Category: N/A, Suspected Domain(s) : N/A (other reasons for verdict). Sender Maturity: 1 month 1 day for domain: nowhere.com
Tue Mar 29 20:20:10 2022 Info: MID 63280 ICID 535151 RID 0 To: <[email protected]>
Tue Mar 29 20:20:10 2022 Info: MID 63280 using engine: SPF Verdict Cache using cached verdict
Tue Mar 29 20:20:10 2022 Info: SPF Verdict Cache cache status: hits = 1, misses = 4, expires = 0, adds = 4, seconds saved = 0.00, total seconds = 0.19
Tue Mar 29 20:20:10 2022 Info: MID 63280 SPF: helo identity [email protected] None
Tue Mar 29 20:20:10 2022 Info: MID 63280 using engine: SPF Verdict Cache using cached verdict
Tue Mar 29 20:20:10 2022 Info: MID 63280 SPF: mailfrom identity [email protected] None
Tue Mar 29 20:20:10 2022 Info: MID 63280 DMARC: Verification skipped (No record found for the sending domain)
Tue Mar 29 20:20:10 2022 Info: MID 63280 DMARC:
Tue Mar 29 20:20:10 2022 Info: MID 63280 Message-ID '<20220330001930.077554@ip-172-31-43-120.us-east-2.compute.internal>'
Tue Mar 29 20:20:10 2022 Info: MID 63280 Subject "test Wed, 30 Mar 2022 00:19:30 +0000"
Tue Mar 29 20:20:10 2022 Info: MID 63280 SDR: Domains for which SDR is requested: reverse DNS host: ec2-3-136-210-164.us-east-2.compute.amazonaws.com, helo: ip-172-31-43-120.us-east-2.compute.internal, env-from: nowhere.com, header-from: nowhere.com, reply-to: Not Present
Tue Mar 29 20:20:10 2022 Info: MID 63280 SDR: Consolidated Sender Threat Level: Neutral, Threat Category: N/A, Suspected Domain(s) : N/A (other reasons for verdict). Sender Maturity: 1 month 1 day for domain: nowhere.com
Tue Mar 29 20:20:10 2022 Info: MID 63280 SDR: Tracker Header : 6243a23a_JVIrZDoIvR4GOwuWdYp/bCtXutJVP2KkniLr+MPFWVUsgiaiJ7VmkqIVm0GO1SqCVMTBcniAH4UyfNxMlcdiLQ==
Tue Mar 29 20:20:10 2022 Info: MID 63280 ready 544 bytes from <[email protected]>
Tue Mar 29 20:20:10 2022 Info: MID 63280 matched all recipients for per-recipient policy ##### URLs in Attachments ##### in the inbound table
Tue Mar 29 20:20:10 2022 Info: ICID 535151 close
Tue Mar 29 20:20:11 2022 Info: MID 63280 interim verdict using engine: CASE spam positive
Tue Mar 29 20:20:11 2022 Info: MID 63280 using engine: CASE spam positive
Tue Mar 29 20:20:11 2022 Info: ISQ: Tagging MID 63280 for quarantine
Tue Mar 29 20:20:11 2022 Info: MID 63280 interim AV verdict using Sophos CLEAN
Tue Mar 29 20:20:11 2022 Info: MID 63280 antivirus negative
Tue Mar 29 20:20:11 2022 Info: MID 63280 AMP file reputation verdict : SKIPPED (no attachment in message)
Tue Mar 29 20:20:11 2022 Info: MID 63280 using engine: GRAYMAIL negative
Tue Mar 29 20:20:11 2022 Info: MID 63280 Outbreak Filters: verdict negative
Tue Mar 29 20:20:11 2022 Info: MID 63280 queued for delivery
Tue Mar 29 20:20:12 2022 Info: RPC Delivery start RCID 3184 MID 63280 to local IronPort Spam Quarantine
Tue Mar 29 20:20:12 2022 Info: ISQ: Quarantined MID 63280
Tue Mar 29 20:20:12 2022 Info: RPC Message done RCID 3184 MID 63280
Tue Mar 29 20:20:12 2022 Info: Message finished MID 63280 done

Looking at the url_rep_client logs:

esa1.hcxxyy-zz.iphmx.com> tail url_rep_client
Tue Mar 29 20:20:10 2022 Info: url_rep_client.beaker_rpc_server : THR: Thread-25: SRC: antispam: MID: 63280: conn-guid:Eme4o07gTA+a85XUX1xgPQ== msg-guid:LtQbVb3sSf2r+kWodvwQ1g==
Tue Mar 29 20:20:10 2022 Trace: url_rep_client.beaker_rpc_server : THR: Thread-25: SRC: antispam: MID: 63280: Urs Req: {'case_ver': '3.11.0', '_request_expiry': 1648599627.466696, 'ver': 1, 'args': ['http://phish.cua.com/test_url_2022032901_rks'], 'mid': 63280, 'msg_guid': 'LtQbVb3sSf2r+kWodvwQ1g==', 'is_outbound': 0, 'ignore_cache': 0, 'connection_guid': 'Eme4o07gTA+a85XUX1xgPQ==', 'id': '12815.1648599610'}
Tue Mar 29 20:20:10 2022 Trace: url_rep_client.beaker_rpc_server : THR: Thread-25: SRC: antispam: MID: 63280: URS GRPC Req: app_info {
  device_id: "420D4F36AAEBC009XXYY-B9E721891234"
  product_family: "esa-ces"
  product_id: "C300V"
  product_version: "14.2.0-468"
}
connection {
  guid: "Eme4o07gTA+a85XUX1xgPQ=="
}
msg_guid: "LtQbVb3sSf2r+kWodvwQ1g=="
url {
  raw_url: "http://phish.cua.com/test_url_2022032901_rks"
  source: URL_LOCATION_EMAIL_BODY
}
no_reputation_block_threshold: true
.
Tue Mar 29 20:20:11 2022 Trace: url_rep_client.beaker_rpc_server : THR: Thread-107: SRC: antispam: MID: 63280: URS GRPC Response: connection_guid: "Eme4o07gTA+a85XUX1xgPQ=="
msg_guid: "LtQbVb3sSf2r+kWodvwQ1g=="
result {
  result {
    threat_level_id: 6
    no_reputation_score: true
  }
}
aup_cat_map_version: 10
threat_cat_map_version: 3
threat_level_map_version: 1
tag_map_version: 1
.
Tue Mar 29 20:20:11 2022 Info: url_rep_client.beaker_rpc_server : THR: Thread-107: SRC: antispam: MID: 63280: URS Batch response received in 0.12s for 1 URLs.
Tue Mar 29 20:20:11 2022 Debug: url_rep_client.beaker_rpc_server : THR: Thread-107: parse_result: [{'no_reputation_score': True, 'threat_level_id': 6}]
Tue Mar 29 20:20:11 2022 Info: url_rep_client.beaker_rpc_server : THR: Thread-107: SRC: antispam: MID: 63280: URLs for msg guid LtQbVb3sSf2r+kWodvwQ1g== with no score to be sent to ECS client: http://phish.cua.com/test_url_2022032901_rks
Tue Mar 29 20:20:11 2022 Trace: url_rep_client.beaker_rpc_server : THR: Thread-107: SRC: antispam: MID: 63280: URS Batch Response: URLs: {'http://phish.cua.com/test_url_2022032901_rks': [{'no_reputation_score': 1, 'aup_cat_id': ['nocat'], 'threat_level_id': 6}]}.
Tue Mar 29 20:20:11 2022 Trace: url_rep_client.beaker_rpc_server : THR: Thread-25: SRC: antispam: MID: 63280: Url Reputation information for the Urls ['http://phish.cua.com/test_url_2022032901_rks']: {'http://phish.cua.com/test_url_2022032901_rks': [{'no_reputation_score': 1, 'aup_cat_id': ['nocat'], 'threat_level_id': 6}]}.
Tue Mar 29 20:20:11 2022 Trace: url_rep_client.beaker_rpc_server : THR: Thread-25: SRC: antispam: MID: 63280: Reply from url rep client: {'ursv2_response': 1, 'reply': [[{'no_reputation_score': 1, 'aup_cat_id': ['nocat'], 'threat_level_id': 6}]], 'unscannable_errors': []}
Tue Mar 29 20:20:11 2022 Info: url_rep_client.beaker_rpc_server : THR: Thread-25: SRC: antispam: MID: 63280: Reply from url rep client: total:1 passed:1 failed:0 unscanned:0 allowed:0 in 0.17s
Tue Mar 29 20:20:11 2022 Info: url_rep_client.beaker_rpc_server : THR: Thread-26: SRC: work_queue: MID: 63280: conn-guid:Eme4o07gTA+a85XUX1xgPQ== msg-guid:LtQbVb3sSf2r+kWodvwQ1g==
Tue Mar 29 20:20:11 2022 Trace: url_rep_client.beaker_rpc_server : THR: Thread-26: SRC: work_queue: MID: 63280: Urs Req: {'origin': 'work_queue', '_request_expiry': 1648599627.925639, 'source': 3, 'args': ['http://phish.cua.com/test_url_2022032901_rks'], 'mid': 63280, 'msg_guid': 'LtQbVb3sSf2r+kWodvwQ1g==', 'is_outbound': 0, 'connection_guid': 'Eme4o07gTA+a85XUX1xgPQ==', 'id': 66}
Tue Mar 29 20:20:11 2022 Info: url_rep_client.beaker_rpc_server : THR: Thread-26: SRC: work_queue: MID: 63280: 1 cache hits in 0.00s seconds
Tue Mar 29 20:20:11 2022 Trace: url_rep_client.beaker_rpc_server : THR: Thread-26: SRC: work_queue: MID: 63280: Url Reputation information for the Urls ['http://phish.cua.com/test_url_2022032901_rks']: {'http://phish.cua.com/test_url_2022032901_rks': [{'no_reputation_score': 1, 'aup_cat_id': ['nocat'], 'threat_level_id': 6}]}.
Tue Mar 29 20:20:11 2022 Trace: url_rep_client.beaker_rpc_server : THR: Thread-26: SRC: work_queue: MID: 63280: Reply from url rep client: {'ursv2_response': 1, 'reply': [[{'no_reputation_score': 1, 'aup_cat_id': ['nocat'], 'threat_level_id': 6}]], 'unscannable_errors': []}
Tue Mar 29 20:20:11 2022 Info: url_rep_client.beaker_rpc_server : THR: Thread-26: SRC: work_queue: MID: 63280: Reply from url rep client: total:1 passed:1 failed:0 unscanned:0 allowed:0 in 0.00s

Remediation

Remember to look in your Remediation logs to see the activity once it's triggered:

esa1.hcxxyy-zz.iphmx.com> tail remediation
Wed Mar 30 23:26:09 2022 Info: MID: 6 Message deleted successfully from [email protected] mailbox.
Wed Mar 30 23:26:09 2022 Info: MID: 6 Remediation succeeded with `bce_demo_com` profile for recipient [email protected].
Wed Mar 30 23:26:09 2022 Info: MID: 7 Message deleted successfully from [email protected] mailbox.
Wed Mar 30 23:26:09 2022 Info: MID: 7 Remediation succeeded with `bce_demo_com` profile for recipient [email protected].

This will match up with the updated mail_logs and Message Tracking for the MID:

esa1.hcxxyy-zz.iphmx.com> tail mail_logs
...
Wed Mar 30 23:26:09 2022 Info: Message 6 was processed due to URL retrospection by Mailbox Remediation with 'Delete' remedial action for recipient [email protected]. Profile used to remediate: bce_demo_com Remediation status: Remediated.
Wed Mar 30 23:26:09 2022 Info: Message 7 was processed due to URL retrospection by Mailbox Remediation with 'Delete' remedial action for recipient [email protected]. Profile used to remediate: bce_demo_com Remediation status: Remediated.

Troubleshooting

Review to make sure you:

  1. Have URL Filtering enabled and it's connected to the URL Retrospection service
  2. Have an action set for Mailbox Auto Remediation (MAR) for URL Filtering - Delete, Forward, Forward To and Delete
  3. Have Anti-spam enabled and active on your mail policy
  4. Have a content filter enabled, utilizing URL Reputation for the condition field
  5. Have Account Settings enabled and actively tied to your O365 account
  6. Have Domain Mapping completed as well in Account Settings
  7. Have ESA upgraded to (at least) 14.2.0-468 or newer, and you are running Cisco Secure Email Cloud Gateway