Page Comparison

INITIATED

Once Verity has checked that the URL is properly formed and does not already exist in the database, the request is passed to the Verity classification systems and the status updates to INITIATED.

PROCESSING

The Verity classification system is processing the text and images on the specified page.

PROCESSED

The URL has been processed and the Verity analysis JSON is available. The analysis results have been stored.

ERROR

Page processing has been attempted and failed. The page URL is recorded in the Error Cache for 1 hour.

If another request to process the same URL is received within 1 hour, Verity will return an Error status (unless the ignoreCache flag is enabled).

After 1 hour, the ERROR status is cleared and Verity will process a new request for the URL. 

Several different conditions may result in an ERROR status message:

• Unreachable page.

• A processing module has returned a value other than a success status code.

NOT_SUPPORTED

The language of the page is not supported (see Language Support Grid ). This status message may also be returned if Verity is unable to process the requested website.

INSUFFICIENT_CONTENT

Verity’s content extraction processes cannot extract sufficient relevant content from a page to adequately perform classification tasks across text and imagery.

INVALID

PAGE_CONTENT_CLASSIFICATION_FAILED

Denotes the failure of the classification process, is usually the client facing generic error code for more granular internal error codes to ease for debugging purposes.

Example Invalid Request:

{

500 - Internal Server Error

PAGE_CONTENT_EXTRACTION_FAILED

A generic error code denoting the failure of the PCE from downloading all the contents of a page successfully. This code is usually saved in the DB along with a more granular error code describing the failure reason of PCE and the status.

If this is the sole code present for a page in DB, it means that tapas has failed to send any value for the expected status field.

Example Invalid Request:

{
 "dataAvailable": false,
 "status": "ERROR",
 "pageUrl": "https://www.gumgum.com", "uuid": "3cd487b5-5715-4173-bfc8-e63d6d6533c0",
 "errors":[
  {
   "reason": "PAGE_CONTENT_EXTRACTION_FAILED"
  }
 ] 
}

422 - Unprocessable Entity

PAGE_CONTENT_EXTRACTION_FAILED_WITH_4XX

This error code denotes that the PCE which is responsible for downloading the contents of a page is facing issues that are mostly from the client side.

On receiving this code the client needs to ensure that the correct values of the URL and the necessary authorisation header values are passed in as the request parameters.

Example Invalid Request:

{
 "dataAvailable": false,
 "status": "ERROR",
 "pageUrl": "https://www.gumgum.com",
 "uuid": "3cd487b5-5715-4173-bfc8-e63d6d6533c0",
 "errors": [
  {
   "reason": "PAGE_CONTENT_EXTRACTION_FAILED"
  },
  {
   "reason": "PAGE_CONTENT_EXTRACTION_FAILED_WITH_4XX"
  }
 ] 
}

422 - Unprocessable Entity

PAGE_CONTENT_EXTRACTION_FAILED_WITH_403_FORBIDDEN

This error code primarily denotes that the PCE which is responsible for downloading the contents of a page is being denied by the host site from doing so mostly due to missing authorisation credentials.

When a client sees this code, it needs to make sure that the correct request parameters and the necessary header values are being passed to

Verity while sending the requests.

Example Invalid Request:

{
 "dataAvailable":false,
 "status": "ERROR",
 "pageUrl": "https:www.gumgum.com",
 "uuid": "3cd487b5-5715-4173-bfc8-e63d6d6533c0",
 "errors": [
  {
   "reason": "PAGE_CONTENT_EXTRACTION_FAILED"
  },
  {
   "reason": "PAGE_CONTENT_EXTRACTION_FAILED_WITH_403_FORBIDDEN
  }
 ]
}

422 - Unprocessable Entity

PAGE_CONTENT_EXTRACTION_FAILED_WITH_404_NOT_FOUND

This error code denotes that the PCE which is responsible for downloading the contents of a page is unable to find the webpage for the requested URL.

On receiving this code the client needs to ensure that the correct values of the URL and the necessary authorisation header values are passed in as the request parameters.

Example Invalid Request:

{
 "dataAvailable": false,
 "status": "ERROR",
 "pageUrl": "https:www.gumgum.com",
 "uuid": "3cd487b5-5715-4173-bfc8-e63d6d6533c0",
 "errors": [
  {
   "reason": "PAGE_CONTENT_EXTRACTION_FAILED"
  },
  {
   "reason": "PAGE_CONTENT_EXTRACTION_FAILED_WITH_

422 - Unprocessable Entity

PAGE_CONTENT_EXTRACTION_FAILED_WITH_

5XX

This error code denotes that the PCE which is responsible for downloading the contents of a page is

experiencing internal issues.

 On receiving this error code

, the client may retry with the request again after a while.

Example Invalid Request:

422 - Unprocessable Entity

PAGE_CONTENT_EXTRACTION_FAILED_WITH

_500_INTERNAL_SERVER_ERROR

This error code denotes that the PCE

is experiencing errors while trying to download the contents of a page

.

On receiving this error code, the client may retry with the request again after a while.

Example Invalid Request:

{
 "dataAvailable": false,
 "status": "ERROR",
 "pageUrl": "https://www.gumgum.com",
 "uuid": "3cd487b5-5715-4173-bfc8-e63d6d6533c0",
 "errors": [
  {
   "reason": "PAGE_CONTENT_EXTRACTION_FAILED"
  },
  {
   "reason": "PAGE_CONTENT_EXTRACTION_FAILED_WITH_500_INTERNAL_SERVER_ERROR"
  }
 ] 
}

422 - Unprocessable Entity

PROCESSING

The Verity classification system is processing the text and images on the specified page.

PROCESSED

The URL has been processed and the Verity analysis JSON is available. The analysis results have been stored.

ERROR

Page processing has been attempted and failed. The page URL is recorded in the Error Cache for 1 hour.

If another request to process the same URL is received within 1 hour, Verity will return an Error status (unless the ignoreCache flag is enabled).

After 1 hour, the ERROR status is cleared and Verity will process a new request for the URL. 

Several different conditions may result in an ERROR status message:

• Unreachable page.

• A processing module has returned a value other than a success status code.

NOT_SUPPORTED

The language of the page is not supported (see Language Support Grid ). This status message may also be returned if Verity is unable to process the requested website.

INSUFFICIENT_CONTENT

Whenever the number of characters in the text associated with the page is less than 50 characters, the further processing of page ceases and the status is saved in the DB.

Example Invalid Request:

{
 "dataAvailable": false,
 "status": "INSUFFICIENT_CONTENT",
 "pageUrl": "https://gumgum.com,
 "uuid": "2fa03d1f-ac16-4e2a-bf24-85135318ed05"
}

422 - Unprocessable Entity

INVALID

The HTTP URL request may be malformed, for example:

• Incomplete URL.

• Missing HTTP header.

• Invalid domain-specific information.

Example Invalid Request:

{
  "dataAvailable": false,
  "Status":  "INVALID", 
  "pageUrl": "://gg.invalid/1"
  "uuid": "73146868-8d77-4bf0-8e89-eeb9d8e04cb2"
}

URL_CANNOT_EXCEED_2048_BYTES

This error code shows that the URL being passed in the request is greater than 2048 bytes and this is deemed to be invalid as this is the limit of Dynamo Db’s partition key’s size limit.

One action client can take on seeing this error code is to ensure that the URL is being stripped from possible query parameters.

Example Invalid Request:

{
 "data Available":false,
 "status": "INVALID",
 "pageUrl": "https://gumgum.com?
q=yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy........yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
 "errors":[
  {
   "reason": "URL_CANNOT_EXCEED_2048_BYTES"
  }
 ] 
}

400 - Bad (Invalid) request.

URL_MALFORMED

This error code denotes that the URL being requested cannot be associated to a standard RFC 2396 value.

Example Invalid Request:

{
 "dataAvailable": false,
 "status": "INVALID",
 "pageUrl": "https://www.gum. gum.com", 
 "errors": [
  {
   "reason": "URL_MALFORMED"
  }
 ] 
}

400 - Bad (Invalid) request.

URL_MISSING_HTTP_PROTOCOL

This error code shows that the requested URL is not adhering the HTTP protocol.

The client should ensure that the URL is beginning with `http://' or ‘https://' .

Example Invalid Request:

{
 "dataAvailable": false,
 "status": "INVALID",
 "pageUrl": "htt://www.gumgum.com",
 "errors": [
  { 
   "reason": "URL_MISSING_HTTP_PROTOCOL"
  }
 ] 
}

400 - Bad (Invalid) request

.

URL_MUST_NOT_BE_EMPTY

This error code denotes that the request is missing the URL. 

Example Invalid Request:

{
 "dataAvailable": false,
 "status": "INVALID",
 "errors": [
  { 
   "reason": "URL_MUST_NOT_BE_EMPTY"
  }
 ] 
}

400 - Bad (Invalid) request

.