Phishing Disruption Service API

The CERT NZ Phishing Disruption Service API (formerly named the Threat Intelligence Feed API) enables the efficient disruption of high volumes of phishing. Scams and phishing are the most prevalent incident types reported to CERT NZ affecting large number of New Zealanders (see CERT NZ’s Quarterly Reports). The API provides two types of actionable indicators of malicious activity – those authored by CERT NZ, and those republished from third parties. The API is built using STIX 2.0 and TAXII 2.0 standards for threat intelligence feed delivery.

CERT NZ publishes Indicators of Compromise (IOCs) via the API. Consumers can then monitor for these indicators in their environment, block them, and return sightings of the IOCs to CERT NZ to help further disrupt malicious actors.

How Phishing Disruption works

Using the API

Follow these steps to get up and running:

Check that your organisation is eligible to use the API

Read the terms and conditions and the eligibility criteria to understand the requirements for gaining access to the API.

Read the technical details

View the API definition in our developer portal here. You can also download the definition for viewing in Swagger Editor, SwaggerHub, or other tools.

Subscribe to the API

Email the appropriately signed MBIE and CERT NZ API access agreements and supporting information as detailed in the eligibility criteria section to the API support team.

Log in and subscribe to the Phishing Disruption Service API Product. Your subscription request will be considered by CERT NZ. Our API support team may request further information before the subscription can be approved.

Please note: The CERT NZ API does not use standard TAXII 2.0 authentication. Please see the TAXII 2.0 Clients and Authentication section for more information.

Generate an API subscription key and call the API

For full details on how to generate an API subscription key, and optionally an OAuth2 bearer token for additional security, please see here.

Once your subscription has been approved you will be able use your API credentials to successfully call the API. Use the sandbox key and sandbox endpoints in your software development. Once you've completed testing simply use your production keys to access the live service.

Specifications and Requirements

TAXII 2.0 Clients and Authentication

Although the Phishing Disruption Service is backed by a TAXII server implementation, the authentication mechanism used by the MBIE API platform will likely pose compatibility issues for TAXII clients.

If an API subscriber has a client implementation that supports code changes or customisations, they may be able to add support for the MBIE API Platform authentication. Below is an example of the reference Python TAXII client with a custom request authentication class to handle the MBIE API authentication and print a list of the collections.

import requests.auth
from taxii2client.v20 import ApiRoot
class MBIE_API_Auth(requests.auth.AuthBase):
  def __init__(self, subscriber_key):
    self.subscriber_key = subscriber_key
  def __call__(self, r):
    r.headers['Ocp-Apim-Subscription-Key'] = '{}'.format(self.subscriber_key)
    return r
apiroot_url = 'https://api.business.govt.nz/sandbox/certnz/phishing-disruption/v2/partners/'
apiauth = MBIE_API_Auth('Subscriber Key From API Portal')
apiroot = ApiRoot(url=apiroot_url, auth=apiauth)
for collection in apiroot.collection:
  print('{} - {}'.format(collection.id, collection.description))

If the API subscriber is unable to modify the client implementation directly, they may need to create a proxy or gateway that handles the authentication.

For cloud environments, a subscriber could define an API gateway matching the definition of the Phishing Disruption Service, but with backend or transformation rules to add the necessary authentication header. When setting up development or server environments, a subscriber could use an Apache or nginx proxy configuration.

TAXII clients that otherwise don’t support the MBIE API Platform authentication can then be configured to use the gateway/proxy and the authentication handled transparently to the client.

STIX 2.0

Organisations must have STIX 2.0 and TAXII 2.0 capability, or a way to translate this data in to their systems. The STIX 2.0 standard defines a consistent yet flexible schema to describe entities within the threat intelligence landscape.  There are objects for:

  • indicators

  • identity

  • sightings

Identity Objects

Within the STIX 2.0 specification:

Identities can represent actual individuals, organizations, or groups (e.g. ACME Inc.) as well as classes of individuals, organizations, or groups (e.g. the finance sector).

Each indicator will have CERT NZ as the Identity. Sightings are expected to be returned with the subscriber identity.

Indicator Objects

Within the STIX 2.0 specification:

Indicators contain a pattern that can be used to detect suspicious or malicious cyber activity. For example, an Indicator may be used to represent a set of malicious domains and use the STIX Patterning Language (STIX™ Version 2.0. Part 5: STIX Patterning) to specify these domains.

CERT NZ creates indicator objects for each unique URL, FQDN, or email that is published in the API.  These objects have valid_from and valid_until dates which specify how long the indicators should be considered as valid intelligence.  CERT NZ considers indicators valid for 7 days unless modified or revoked earlier.

Sighting Objects

Within the STIX 2.0 specification:

A Sighting denotes the belief that something in cyber threat intelligence (e.g. an indicator, malware, tool, threat actor, etc) was seen. Sightings are used to track who and what are being targeted, how attacks are carried out, and to track trends in attack behaviour.

API partner organisations send CERT NZ Sightings to convey information about when and where indicators are seen. 

Relationship between STIX 2.0 objects in the API

Relationship between STIX objects

Revocation and Expiry

API consumers are responsible for tracking the validity of indicators and modifying any disruption activities when indicators cease to be valid (such as blocking). An indicator ceases to be valid when its valid_until date passes, or it is revoked. 

Organisations need to store the valid_until date for the indicators they consume and remove them from any block lists and stop any network disruption activities when the valid_until date is reached. 

An indicator might be revoked when CERT NZ becomes aware of the fact that the issue that caused the indicator to be published has been resolved. For example, a website may unwittingly be hosting a phishing kit and as a result the URL or domain of that website is published as an indicator in the API. If the website administrator resolves the issue and provides sufficient evidence to CERT NZ, a new version of that indicator will be published in the API with a revoked property set to 'true'.   

Resources available

  1. Open API definition: for technical people to understand how to consume the API. Refer to the developer portal page here.

  2. STIX 2.0 specification: STIX defines how objects relate to one another. For example a single ‘campaign’ might reference a number of ‘malware’ types being deployed. The initial CERT NZ offering will include ‘indicators’, ‘identity’ and ‘sightings’ STIX objects. Explanations needed to understand the service:

  3. TAXII 2.0 specification: TAXII protocol is a TLS secured RESTful API which defines a standard interface for sharing STIX objects. It is expected that over time it will be common for information security products (such as IDS/IPS systems) to integrate with the TAXII service. This should be read in conjunction with CERT NZ’s Swagger file.

How are CERT NZ authored indicators created?

  1. Phishing email received by natural person

  2. They report it to CERT NZ

  3. CERT NZ extracts the indicators out of the message

  4. CERT NZ analyses, filters and enriches the indicators

  5. CERT NZ publishes the indicators via the API to end users

  6. Indicators are used by partners to identify and block threats

Proactive incident response

Some of the phishing indicators identified by CERT NZ will be New Zealand systems. Because of the nature of the phishing incident, the system owners may not know they are affected. When CERT NZ identifies an affected system in New Zealand, our incident response team will reach out to the owner and help them resolve their issue.

API rate limiting

There is a limit of 1,000 requests per user per minute. When the limit is exceeded you will receive an HTTP status 429 response similar to this:

Reference information

CERT NZ STIX detail 

Notes:

  • All timestamps are valid RFC 3339-formatted timestamps using the format YYYY-MM-DDTHH:mm:ss[.s+]Z where the "s+" represents 1 or more sub-second values. The brackets denote that sub-second precision is optional.  The following are valid examples:

    • "valid_until": "2023-04-05T05:32:29.281Z"

    • "modified": "2022-11-24T06:38:02.124Z",

    • "valid_from": "2023-01-23T01:55:24Z"

  • All timestamps are in UTC time and will have millisecond precision up to three decimal places.

  • All STIX objects have an RFC-4122-compliant Version 4 UUID, annotated in this documentation as <UUID>.

  • STIX objects allow a list of marking definitions to be defined for each object, but the specification does not specify how multiple marking definitions are to be interpreted. For clarity, all STIX objects will have one or zero marking definitions, which apply to the whole object.

Indicators from CERT NZ

Bundles

STIX indicators will be published together in a STIX bundle:

The indicators will be included in the objects list.

Indicators 

An indicator is an Object that contains a pattern that can be used to detect malicious cyber activity. It also contains information on the creator, the validity period and whether it has been revoked or not.

It is important that consumers respect the valid_until and revoked properties of this object. Several important signals are communicated via these properties:

  • Indicator should be expired (valid_until)

  • Indicator validity has been extended (valid_until)

  • Indicator has been revoked (revoked)

While rare, revoked Indicators imply an issue has been identified with an indicator such as a high false-positive rate. These should be processed in a timely manner and will always appear with appropriate use of the added_after query.

While the STIX 2.0 patterning language allows for rather complex patterns, CERT NZ will only make use of basic, atomic patterns for common indicator types. The current exhaustive list of patterns are:

  • [domain-name:value='bad.example.com']

  • [ipv4-addr:value='10.1.2.3']

  • [url:value='http://bad.example.com/bad.exe']

STIX 2.0 Part 2: 2.5 Indicator
STIX 2.0 Part 5: STIX Patterning

Each indicator will contain a pattern for a specific IOC. This example is for the URL with address http://example.com/bad:

Note:

  • See below for examples of marking definitions and identities.

  • STIX patterns are used in the pattern field. CERT NZ will use the following pattern types:·          

    o    url

    o    domain-name

    o    ipv4-addr

    o    ipv6-addr

  • While the standard allows for complex patterns CERT NZ will only use one pattern per indicator.

Marking Definition

A Marking Definition Object represents a specific marking. These typically represent handling or sharing requirements – for example the Traffic Light Protocol (TLP). Data markings define restrictions, permissions and other guidelines for how marked objects can be used and shared . Marking definitions will also be included in a bundle's objects list. STIX 2.0 explicitly defines all TLP markings.

Note: The STIX 2.0 defines the implementation of TLP – these do not necessarily align to the TLP standard.

This API exclusively makes use of these predefined markings and they are all included in the Phishing Disruption Feed. At this stage all Phishing Disruption API indicators are shared at TLP Amber.


STIX 2.0 Part 1: 4.1 Marking Definition
STIX 2.0 Part 1: 4.1.4 TLP Marking Object Type

Identity

Identities uniquely identify an individual, group, organisation, etc. This Object captures basic identifying and contact information. It is primarily used in STIX to provide information relating to Object creators.

The only identity currently included in the Threat Intelligence Feed Indicators Collection is the CERT NZ identity. All indicator objects will have a created_by_ref set to CERT NZ’s identity.

When returning telemetry, all consumers must include an identity object with every telemetry submission which corresponds to their organisation's identity.
STIX 2.0 Part 2: 2.4 Identity

Sightings

A sighting is a special relationship in STIX, a defines a relationship between an indicator (what was sighted), an identity (who created it), and any external_references (who originally sighted it). It is technically a STIX 2.0 Relationship rather than a STIX 2.0 Object. It represents a potential detection of an item of threat intelligence. It links the following entities together:

  • What was sighted (i.e. the indicator, sighting_of_ref)

  • Who sighted it (i.e. an identity, created_by_ref`)

  • When it was sighted ( i.e. the first_seen and last_seen fields)

A Sighting alone contains enough properties to fully represent the concept of Telemetry as defined by the CERT NZ API Access Agreement.

CERT NZ expects to receive Sightings in an unaggregated fashion. That is, a Sighting relationship should be included for every single detection of an indicator. In practice, this is generally easier for consumers because there is no need to buffer and aggregate sightings over a period of time.

STIX 2.0 Part 2: 3.2 Sighting

A sighting without an external_references field denotes that it was originally created by CERT NZ.

Telemetry to CERT NZ

To ensure the quality of the feed, CERT NZ need consumers to return telemetry when a detect or disrupt action has been taken based on the indicators in the feed.

This is the mechanism through which telemetry is submitted to the CERT NZ. The client must generate and provide valid STIX 2.0 Objects to include in the TAXII 2.0 bundle.

While the Phishing Disruption TAXII 2.0 API must support submission of any STIX 2.0 Object, only the following Object types constitute telemetry:

  • Identity

  • Sighting

See the API definition for more information about what data needs to be included in each of the above Objects.

Example request

The steps below will return a list of phishing indicators added since 1 June 2019. Environment may be either "sandbox" for testing or "gateway" for production use.

1. Get the list of available collections

Endpoint
GET https://api.business.govt.nz/{environment}/certnz/phishing-disruption/v2/partners/collections/
Headers
Accept: application/vnd.oasis.stix+json; version=2.0
Content-Type: application/vnd.oasis.stix+json; version=2.0
Ocp-Apim-Subscription-Key: {{subscriber-key}}

2. The response will include details of the collections available from the CERT NZ Threat Feed API, including the collection id values to use in other API calls. There is currently one collection available, the Phishing Indicator Collection.

3. Use the collection id for the Phishing Indicator Collection in a second call to get the list of objects. The added_after and match[type] query parameters are set to limit the results to the STIX objects of interest in this example. The Range header is set to return the first 20 objects.

Endpoint
GET https://api.business.govt.nz/{environment}/certnz/phishing-disruption/v2/partners/collections/7605c355-ff43-4dc5-8f21-86f4b909bbf1/objects/?added_after=2023-01-09T00:00:00Z&match[type]=indicator
Headers
Accept: application/vnd.oasis.stix+json; version=2.0
Content-Type: application/vnd.oasis.stix+json; version=2.0Range: items 0-19
Ocp-Apim-Subscription-Key: {{subscriber-key}}

4. The response body will include the objects that meet the search criteria. The response status will be 206 Partial Content when the results page size is smaller than the number of available results, or when the request has included a Range header. The response will include a header Content-Range that states the number of objects returned out of the total results available, e.g. Content-Range: items 0-19/3976.

Example bundle

The following bundle is an example of what can be expected from the CERT NZ API. It contains two sightings and their associated indicators. One of the sightings comes from the OpenPhish upstream provider, the other is authored by CERT NZ itself (as evidenced by the lack of external_references section in the sighting object.

The CERT NZ identity object is also included.

Example expiry-check

As mentioned in the Polling, aging and revoking section of the terms and conditions, prior to expiring an indicator you should check to see if CERT NZ has extended its expiry. The three possible outcomes are shown here. In all cases the requester has an indicator (indicator--0be8387e-51e7-4ca1-92e9-5f20e8e19fff ) which they intend to expire at the following time: "2023-03-15T05:51:05.05Z".

Request details of an indicator which has already been expired.

GET https://api.business.govt.nz/{environment}/certnz/phishing-disruption/v2/partners/collections/7605c355-ff43-4dc5-8f21-86f4b909bbf1/objects/indicator--0be8387e-51e7-4ca1-92e9-5f20e8e19fff/?match[version]=last

1: The indicator has already expired

In this case the requester can go ahead and expire their indicator (and associated sightings immediately).

2: There has been no expiry extension 

As can be seen above, the valid_until time is the same as the one the requester already had.  The requester can wait until that time and expire their indicator (and associated sightings immediately). 

3: CERT NZ has extended the expiry on this indicator

As can be seen in the valid_until field above, CERT NZ has extended the expiry for this indicator. The requester should not expire this indicator, instead they should request all versions of this indicator and ensure that their local copies are up to date.

 

API access conditions

Terms and conditions

Eligible organisations need to sign up to MBIE’s general API Access Agreement, as well as the CERT NZ API Access Agreement that describes the terms and conditions specific to CERT NZ’s Phishing Disruption API. Refer to the eligibility criteria section for details. Key obligations in the CERT NZ API Access Agreement are summarised below.

Data protection and confidentiality

All data in the API is shared under the Traffic Light Protocol (TLP) with specific rules defined in the CERT NZ API Access Agreement to ensure data protection requirements are met. These include:

  • Use of information in the API for commercial gain i.e. the information can be used with your customers, but you can’t sell it.

  • Solicitation - the data can’t be used for any other purpose e.g. soliciting sales.

  • You can’t use information in the API to establish any person’s identity or gather personal information.

Reporting

Organisations need to report information back to CERT NZ such as telemetry or aggregate metrics, false positives, and sightings such as confirmed compromised URLs, IOCs.

It is likely that CERT NZ will update the Access Agreement in future to make this compulsory once more methods of reporting and products have been developed.

Acknowledgement or identification of CERT NZ

Where data is sourced from the CERT NZ API, API Consumers may publically reference that they have access to this data, and they can use it as part of a suite of decision-making tools to protect their customers and staff.

However you can’t attribute any action you take based on information or indicators provided by CERT NZ, to CERT NZ. This is because the actions you take based on the information we provide should involve your own due diligence and decision processes. Read the 'Acknowledgement Guidelines' schedule in the CERT NZ API Access Agreement for more information.

Polling, aging and revoking

You need to poll the API at least hourly for new and updated data, and age-off data after 7 days or as otherwise stated in the specs.

CERT NZ adds most new indicators to the API feed hourly. There is little benefit in querying the API more regularly than every 60 minutes. It is possible to use the added_after TAXII query to request indicators which have been newly created or revoked since the last time you queried the API.  In this way if you miss a query for whatever reason (your client was down for example), you are able to catch up later.

Revoking and Validity extensions

By default, new indicators are valid for 14 days.  If CERT NZ continues to see additional instances of this indicator after half of this validity period (7 days), it will extend the lifetime of the indicator by another 14 days. At the point an indicator expires, it will be removed from the API feed and should not be considered actionable intelligence.

An example of the lifelines of three indicators is shown below to illustrate these concepts.

Picture

Updates to an indicator’s expiry time are not automatically visible by using the added_after functionality mentioned above. This is because the TAXII standard specifies the added_after filter to be relative to the datetime the indicator was added to the TAXII server, not the most recent update to an indicator. 

To ensure that customers are not expiring indicators that CERT NZ has extended, specific indicators should be queried for their latest version just before they are expired. That will return either:

  • no result (CERT NZ has already expired the indicator and you should do the same);

  • a version with the same expiry date you currently have (feel free to expire the indicator at the time specified); or 

  • a version with a new expiry (you should update your version)

In these cases, the following approach works well:

  • Periodically query your own database for indicators due to expire (e.g. in next five minutes)

  • For each expiring indicator, request manifest from CERT NZ TAXII server

  • If a new version exists in manifest, request object from CERT NZ TAXII Server

  • If no new version exists, remove indicator from any detection or disruption capabilities

CERT NZ places a high importance on a revoked Object being acted on in a short period of time, these notifications will always appear in an added_after query. This is because CERT NZ deletes the revoked Object and adds it to the feed again (thereby giving a new date_added time).

If an indicator appears with `revoked` set to true for any given poll, it should be immediately removed from any detection or disruption capabilities.

It is possible CERT NZ adds an indicator to the API feed and the underlying issue is remediated well before the 14 day expiry.  An example may be the website of a small online New Zealand business being compromised and hosting a phishing site. At the same time the indicator for this site is added to the API feed, CERT NZ incident responders will reach out to the business and encourage them to fix their issue and secure their website.  This may well take less than the 14 day expiry of the indicator that CERT NZ published on the feed.  In this case the business may request that CERT NZ revoke the indicator to encourage API feed consumers to allow traffic back to their website. 

As an API feed consumer, you must revoke or remove any block or action from your feed within 24 hours if notified or flagged within the data (expiry field is set).

Off-boarding

For security reasons, you are responsible for revoking authorised user access if their employment or relationship with your organisation ceases.

If you stop using the API entirely, you’ll need to revoke any active mitigations (such as blocking traffic) taken based on CERT NZ indicators, and remove all API functionality.

Read the full CERT NZ API Access Agreement to find out more.

CERT NZ API eligibility criteria

Overview

The information provided under the CERT NZ Phishing Disruption API access agreement is sensitive and may contain information that could identify organisations affected by a cyber security threat in New Zealand. CERT NZ has developed eligibility criteria to enable partners that can make effective use of the data, while also excluding the types of actors whose ability to make use of the data is not sufficient to justify providing access to this commercially sensitive information.

CERT NZ takes the sensitivity, privacy and disclosure of information seriously, as detailed in our Privacy and Disclosure Statement.

Purpose

The Phishing Disruption API data will only be provided to eligible partners where CERT NZ determines that they can assist with the following defensive cyber security activities:

  • Help affected organisations remediate compromises, or

  • Protect their customers or staff from being exposed to cyber threats (such as restricting customer access to a phishing website), or

  • Provide demonstrable security benefits, or threat mitigation for New Zealand organisations and New Zealanders.

If information provided under this access agreement is used incorrectly, or in breach of the CERT NZ API Access Agreement, it could have a detrimental impact on potentially affected organisations.

Access to the API is not a prerequisite to work with CERT NZ, and information sharing outside of the API mechanism regularly occurs with organisations reporting or receiving information in order to respond to or mitigate specific cyber security threats and incidents.

Types of Eligible Organisations

To ensure that the sensitive data provided via the API is limited to those able to act on and protect the sensitivity of it, CERT NZ will only consider the following types of organisation for their eligibility to enter into an access and use agreement to receive this confidential information:

  • Internet Service Providers (ISPs)

  • Security Service Providers (SSPs)

  • Hosting Service Providers (HSPs)

  • Organisations engaged in cyber security research and development (R&D)

  • Large Network Operators (LNO)

Collectively, it is expected that eligible types of organisations will be able to provide significant coverage and protection for New Zealand against cyber security threats.

Eligibility Consideration Factors

CERT NZ will consider the following factors when determining eligibility to enter an Access Agreement:

  • That the organisation falls under the types of organisation deemed eligible under this criteria;

  • The organisation’s reach and customer base (if used to protect customers), staff numbers if deployed internally within that partners network;

  • Their ability to consume and use the data provided via the CERT NZ API; How the organisation proposes to use the information provided, and the ability to provide information back to CERT NZ on sightings (as outlined in the API access agreement);

  • That the organisation agrees to the API terms and conditions; and

  • Any other factor of material relevance

CERT NZ retains sole discretion to determine eligibility.

Eligibility Applications

Organisations that wish to apply to gain access to the CERT NZ Phishing Disruption API must supply the following information via the MBIE API support team at apisupport@business.govt.nz. The API support team will contact you to request details if this information has not been received at the time of the API subscription request:

  1. The organisation name and service description

  2. The email address for the API subscription user profile must be a generic contact, not an individual (e.g. security@organisation.co.nz)

  3. The type of organization category they belong

  4. The current and future ability to consume the CERT NZ API

  5. How they intend to use the information consumed if eligible

  6. Any other information of relevance to their application

  7. A signed MBIE API Access Agreement

  8. A signed CERT NZ API Access Agreement

Review

CERT NZ may review, consult and amend the eligibility criteria at any time, including wider or reducing the types of organisations able to access it, and any factors relevant to maintain the integrity of the Phishing Disruption API system. Any changes will apply only to new applicants; existing consumers of the API will be subject to their agreed CERT NZ API Access Agreement.