WoAG Occupation Coding Service User Guide

Before accessing the service, you will need to register for the coding service (see Registration). You will also need to consider the following:

• What you need to set up to pass the API packet to your API endpoint (url).
• What data you are going to code.
• Whether you need single record coding, small batch coding (up to 300 records), or large batch coding.
• Whether you need to reformat your data (for example, batch data will need to fit specific formats, and you may want to add record identifiers to map back to your dataset).

See also Coding Service formats, and the following sections in Using the service: Important context for model use, Tips for getting the best predictions out of the Coding Service, and Review coding outputs.

The use of the Coding Service API is governed by the Coding Service Terms of Use (which includes Service Level Expectations). All API users will be required to accept these Terms of Use prior to gaining access to the service. 

Users requesting access to the service must be appropriately authorised to accept the Terms of Use on behalf of their organisation. 

To register for the service and request client credentials, please read and accept the Terms of Use and complete the Registration Form. 

 Once you have registered, your ABS contact will email you: 

• a client identifier, 'clientID', and
• a client secret, 'clientSecret'. These must be kept confidential as they are used when authenticating your requests. 

The ABS will monitor registrations for usage, and users will be notified via email if their access is under review.

An authentication token is required to use the Coding Service. This is a unique, time-limited access key which is used to authenticate all API calls to the service.

To get an authentication token, you must first call the service’s authentication endpoint with an authorisation header. More details are available in the AWS documentation, but the key input parameters are described below.

Request syntax

POST /oauth2/token HTTP/1.1
Host: string 
Content-Type: application/x-www-form-urlencoded
Authorization: string
{
    grant_type: "client_credentials"
}

See Integration script examples for an example PowerShell script to request an authentication token. 

Request header parameters

Host: 
     The host name for your chosen API. This will be either https://partner-coder.auth.abs.gov.au or
     https://public-coder.auth.abs.gov.au depending on whether you have registered for the partner or the 
     public coding service.

Authorization: 
     Basic authorisation method with a base64 authorisation token (encodedAuthString), computed from the 
     client ID and client secret provided upon registration. 
     encodedAuthString can be computed via the bash command:

     $ echo -n "${clientID}:${clientSecret}" | base64
     Type: String

Response syntax

HTTP/1.1 200 OK
Content-Type: application/json
{
     access_token: "string"
}

Response elements

access_token
     Your unique access token which can be used to authenticate all API calls to the coding service.
     Type: String

Examples

On registering for the coding service, this user was issued with the following:

• ClientID: “client1”
• ClientSecret: “secret123”

encodedAuthString should be the base64 encoding of “client1:secret123” and the entire request is as follows:

(See Integration script examples for an example PowerShell script to request an authentication token.)

To authenticate against the coding API service, you will need to include your access token in the header of any API calls.

The API calls are of a short duration, usually less than a few seconds. When initiated, each call will check the authentication and then continue with the rest of the call. If the call was approved at the start, it will return a response if the timer runs out. 

Asynchronous batch calls may take longer to return results, and you may have to re-authorise to receive the results.

Authorization
     The authorisation token retrieved via the Authentication mechanism.
     Type: String

Host
     The host name for your chosen API. This will be either https://partner-coder.api.abs.gov.au or 
     https://public-coder.api.abs.gov.au depending on whether you have registered for the partner or the
     public coding service.
     Type: String

For optimal performance, please reuse a single session token per hour, and avoid hitting service limits by caching tokens. Single session tokens will be valid for all users interacting with webforms or interfaces during that time. 

• Token Type: AWS Cognito M2M session token.
• Token Validity: 60 minutes.
• Usage Scope: Expect shared use of a single token across all users of a webform/integration/process.
• Storage: Store locally in your backend (e.g., in-memory, file, cache).
• Rate Guidance: Max 1 token request per hour per webform.

Exceeding this limit may result in your service being blocked or degraded. Restrictions are time limited, and requests for authorisation during this time will result in HTTP 403 errors.

Please ensure that any scripts written for others to copy and paste (i.e. for coding single records or running small batches) do not include a token call. Provide authentication scripts separately to be used at need. 

Recommended integration pattern

1. Backend checks for existing token.
2. If token is valid, reuse it.
3. If token is expired or missing, request a new one.
4. Use the token to call the API for all users of the form.

Token re-use flow diagram

Recommended flow for session token re-use across users of a webform.

The coding service uses JSON format for the following services:

• Real-time (synchronous) public coding service
• Real-time partner coding service
• Real-time small batch coding service

It uses JSONL format for the large batch/bulk (asynchronous) partner coding service. 

The GET Data method will return the following for each of the specified services for occupation coding:

• The occupation coder will perform optimally when provided with both a job title and tasks as free text inputs, as this is how the ML training was carried out.
• Large batch (asynchronous) coding requires both the occp_text and tasks_text fields in the call. If input text for one or other of the fields is not available, include the blank field. For example: 
              "occp_text": "sewing machinist"
              "tasks_text": ""
• The coder will not perform as well with just one text field completed (i.e if only the job title or only the task text is entered). If results are unsuccessful, entering more information will help the service make better predictions.
• For synchronous coding, text strings can be a maximum of 100 characters only (a total of 100 characters for combined occupation and task input text entries).
• For asynchronous coding, text strings can be a total of 300 characters for the combined fields.
• The coding service API will not accept custom data queries or query string parameters.
• The service does not recognise classification codes as inputs. While it is not possible to recode a six-digit ANZSCO code to a six-digit OSCA code, datasets with only ANZSCO codes may be recoded to OSCA if the ANZSCO occupation title is reattached to each record. Including more information as a tasks text entry, such as the occupation descriptions from the classification, will give better outcomes.
• The coding service has been trained on English inputs only. The service accepts printable ASCII characters, which includes all English letters and connectives, but excludes certain accents, foreign currency symbols and control characters like file endings or backspace. Including a bad character may result in an 'Invalid request body' error.

The contextual assumption of the input text is that the text relates to and describes a person’s job. The coder is able to recognise a very broad vocabulary and will attempt to code all input text, regardless of context, so users need to ensure a contextual fit between their input data and the coding task being undertaken. 

For example, if a person enters their job text as ‘prisoner’, the Coding Service assumes a context that the job to be coded works with prisoners in some way, and codes to ‘Correctional Officer’. Likewise, the input text ‘student’ codes to ‘Student Services Adviser’. 

The OSCA 2024 model also returns non-classification codes for responses that are not occupations within the scope of the classification:

      code     label
     099900  Not in the labour force nfd
     099988  Inadequately described
     099920  Child/baby
     099930  Invalid pensioner
     099940  Other pensioner
     099950  Housewife/husband
     099960  Retired
     099970  Unemployed/Not work for the dole

(Please note: the ANZSCO 2022 model also available in the Coding Service does NOT include non-classification codes.) 

The service is designed to provide a single occupation code and title for a single record. If multiple jobs per record are entered in the occp_text field, the coder will attempt to code the provided text to a best fit single occupation code at the most detailed level. 

The output will reflect the training data, and will depend on how many times the two jobs were present together in the training data. The Coding Service will default to whatever is most commonly found in the training data.

• If multiple jobs are present, you will need to format each job as a separate request. 

Likewise, if people combine non-classification labels with job titles, such as ‘retired boilermaker’, ‘unemployed postman’, etc, the coder will attempt a best fit code. This may be ‘retired’, or ‘boilermaker’, or something else entirely depending on the context and the amount of times the combination of words appeared in the training data. These records may require review.

For more detail, see Using the service: Important context for model use, Tips for getting the best predictions out of the Coding Service, and Review coding outputs.

The API endpoints and their HTTP methods are outlined below in both the table and the diagram.

Before coding against a topic (classification), you must confirm that the topic is supported by the application. You will also need to record its corresponding uriName for further calls to the API. 

GET /v1/topics HTTP/1.1
Host: string
Content-type: application/json
Authorization: string

The request does not use any URI parameters.

The request does not have a request body.

HTTP/1.1 200 OK
Content-type: application/json
[
    {
        "uriName": "string",
        "fullName": "string"
    }
]

If the action is successful, the service sends back an HTTP 200 response. The API returns an array of Topic objects representing all the coding topics currently supported by the API.

For information about errors selecting topics, see Errors and suggested actions.

Different models are coded against different input formats. If you are using the latest (default) model for your specified topic, you can get the input format via the following mechanism.

If you are using another model, the input format will be provided as part of the list of available models.

GET /v1/topics/{topic}/models/latest HTTP/1.1
Host: string
Content-type: application/json
Authorization: string

The request does not have a request body.

HTTP/1.1 200 OK
Content-type: application/json
{
    "modelId": "string",
    "modelVersion": number,
    "modelReleaseDate": "string",
    "modelType": "string",
    "inputFormat":  [
        "string"
    ],
    "topicStandard": "string",
    "topicVersion": "string"
}

If the action is successful, the service sends back an HTTP 200 response. The API returns a Model object representing the latest model for the given topic, which includes the expected input format.

For information about errors selecting models, see Errors and suggested actions.

Getting the latest occupation model:

To code against a specific machine learning model, you can browse the available models and their input formats by calling this endpoint. 

GET /v1/topics/{topic}/models HTTP/1.1
Host: string
Content-type: application/json
Authorization: string

The request does not have a request body.

HTTP/1.1 200 OK
Content-type: application/json
[
    {
        "modelId": "string",
        "modelVersion": number,
        "modelReleaseDate": "string",
        "modelType": "string",
        "inputFormat": [
           "string"
        ],
        "topicStandard": "string",
        "topicVersion": "string"
    }
]

If the action is successful, the service sends back an HTTP 200 response. The API returns either a SynchronousCodeResponse object or an array of SynchronousCodeResponse objects corresponding to the input records.

For information about errors selecting models, see Errors and suggested actions. 

Listing all ANZSCO models:

The coding service has been designed to apply a classification code and title to a free text entry. The synchronous single record coding feature will enable public facing webforms and other points of data collection to have codes and titles suggested in real time (~1 second). 

• The service will provide multiple responses within a classification category, as long as at least one response has a confidence level above the threshold.
• It may only provide one response - for example, if there is only one six-digit code in the relevant category.
• If no responses are above the confidence threshold, the service will not return any results. 

Small batch coding

A small JSON file of up to 300 text records can also be coded synchronously. 

Note: when you are running a synchronous small batch, the whole packet needs to be syntactically correct. If the syntax fails, the whole batch will fail. As the operation is combined for the whole group of records, none of the records will be able to be coded if there is an error in any record.

This endpoint is used to code a single or small batch of free text records against the specified coding topic, using the latest model for that topic.

Depending on whether you are coding a single record or a small batch of records, your request will follow one of the following formats:

1. Coding a single free text record

POST /v1/topics/{topic}/code HTTP/1.1
Host: string 
Content-type: application/json
Authorization: string
{
    "record": {
        "occp_text": "string",
        "tasks_text": "string"
    },
    "numberOfSuggestions": number
}

2. Coding a small batch of free text records

POST /v1/topics/{topic}/code HTTP/1.1 
Host: string
Content-type: application/json
Authorization: string
{
    "records": [
        {
            "recordId": "string",
            "occp_text": "string",
            "tasks_text": "string"
        },
    ],
    "numberOfSuggestions": number
}

See Integration script examples for example PowerShell single record and small batch coding scripts.

The response of this endpoint will depend on whether your input request contained a single record or a small batch of records.

1. Coding a single free text record

HTTP/1.1 200 OK
Content-type: application/json
{
    "codeStatus": "string",
    "input": {
        "occp_text": "string",
        "tasks_text": "string"
    },
    "result": [
        {
            "codeCategory": "string",
            "codeLabel": "string",
            "codeConfidence": number
        }
    ],
}

2. Coding a small batch of free text records

HTTP/1.1 200 OK 
Content-type: application/json 
[
    {
        "recordId": "string",
        "codeStatus": "string",
        "input": {
            "occp_text": "string",
            "tasks_text": "string"
        },
        "result": [
            {
                "codeCategory": "string",
                "codeLabel": "string", 
                "codeConfidence": number
            }
        ],
    }
]

If the action is successful, the service sends back an HTTP 200 response. The API returns either a SynchronousCodeResponse object or an array of SynchronousCodeResponse objects corresponding to the input records.

For information about synchronous coding errors, see Errors and suggested actions. 

(Also see Integration script examples for example PowerShell single record and small batch coding scripts.)

This endpoint is used to code a single or small batch of free text records against the specified coding topic, using the specified model.

Depending on whether you are coding a single record or a small batch of records, your request will follow one of the following formats:

1. Coding a single free text record against a specific model

POST /v1/topics/{topic}/models/{model}/code HTTP/1.1
Host: string
Content-type: application/json
Authorization: string
{
    "record": {
        "occp_text": "string",
        "tasks_text": "string"
    }
    "numberOfSuggestions": number
}

2. Coding records against a specific model 

POST /v1/topics/{topic}/models/{model}/code HTTP/1.1 
Host: string
Content-type: application/json
Authorization: string
{
    "records": [
        {
            "recordId": "string",
            "occp_text": "string",
            "tasks_text": "string"
        }
    ],
    "numberOfSuggestions": number
}

The response of this endpoint will depend on whether your input request contained a single record or a small batch of records.

1. Coding a single free text record

HTTP/1.1 200 OK
Content-type: application/json
{
    "codeStatus": "string",
    "input": {
        "occp_text": "string",
        "tasks_text": "string"
    },
       "result": [
        {
            "codeCategory": "string",
            "codeLabel": "string",
            "codeConfidence": number
        }
    ],
}

2. Coding a small batch of free text records

HTTP/1.1 200 OK 
Content-type: application/json 
[
    {
        "recordId": "string",
        "codeStatus": "string",
        "input": {
            "occp_text": "string",
            "tasks_text": "string"
        },
        "result": [
            {
                "codeCategory": "string",
                "codeLabel": "string",
                "codeConfidence": number
            }
        ],
    }
]

If the action is successful, the service sends back an HTTP 200 response. The API returns either a SynchronousCodeResponse object or an array of SynchronousCodeResponse objects corresponding to the input records. 

For information about synchronous coding errors, see Errors and suggested actions. 

(Also see Integration script examples for example PowerShell single record and small batch coding scripts.)

In addition to real-time coding of single records and small batches of data, the Coding Service has been designed to code large datasets through asynchronous batching (that is, returning data after a short period of time). 

The asynchronous service can be used for as little as one record, up to millions of records.

Note: Asynchronous batch coding should be used if you need to code or recode a large volume of data. While it is the most efficient method of coding larger datasets, it is not real-time, and may be subject to queueing during high load periods. 

This endpoint is used to create an asynchronous batch inference operation. The API will return a location where you can upload your input file and begin your batch inference operation.

Depending on whether you are specifying a model against which to code your records, your request will follow one of the following formats:

1. Coding records against the latest model

POST /v1/topics/{topic}/batch-code HTTP/1.1
Host: string
Content-type: application/json
Authorization: string

2. Coding records against a specific model

POST /v1/topics/{topic}/models/{model}/batch-code HTTP/1.1 
Host: string
Content-type: application/json
Authorization: string

The request does not have a request body.

HTTP/1.1 200 OK
Content-type: application/json
{
    "requestUploadUrl": "string",
    "operationId": "string",
    "bucketKmsKeyArn": "string"
}

For information about asynchronous coding errors, see Errors and suggested actions.

Once you have created an inference operation, you will need to upload your data to the provided requestUploadUrl. This is a pre-signed HTTP request which is managed by the AWS S3 server, and the expected input is outlined below. 

• Both occp_text and tasks_test fields are required (although one may be empty, indicated by “”).

PUT requestUploadUrl HTTP/1.1
x-amz-server-side-encryption: aws:kms
x-amz-server-side-encryption-aws-kms-key-id: string
{ "recordId": "string", "occp_text": "string", "tasks_text": "string" }
...

HTTP/1.1 200 OK

For information about the errors that are common to all actions, see Errors and suggested actions.

This endpoint is used to check the status of your batch inference job. When the status of your job is complete, the service will return a URL to copy into your web browser to retrieve your coded data.

Depending on whether you are specifying a model against which to code your records, your request will follow one of the following formats. The application backend handles these requests identically, so you don’t need to worry about recording the model which you used when you began the operation.

1. Checking an operation by specifying the topic only

GET /v1/topics/{topic}/batch-code/operations/{operation_id} HTTP/1.1
Host: string
Content-type: application/json
Authorization: string
 

2. Checking an operation by specifying both the topic and model 

GET /v1/topics/{topic}/models/{model}/batch-code/operations/{operation_id} HTTP/1.1
Host: string
Content-type: application/json
Authorization: string

The request does not have a request body.

HTTP/1.1 200 OK
Content-type: application/json
{
    "operationStatus": "string",
    "responseDownloadUrl": "string",
    "error": "string"
}

A state machine indicating the progression of operations is shown below:

For information about asynchronous coding errors, see Errors and suggested actions.

Once your asynchronous inference operation is complete, you can download the output file by accessing (copying into a web browser) the responseDownloadUrl that is provided when you check the status of a complete operation. The same process may be used to view the operation metadata, available at the metadataDownloadUrl.

This is a generic HTTP GET request which is managed by the AWS S3 server, and the expected format is outlined below.

The asynchronous batch coding service outputs a jsonl file with each line corresponding to the record from the original input file. Each line is an AsynchronousCodeResponse object.

If you encounter bugs or have feedback on the service, please report these via the following mechanism:

GET /v1/security.txt HTTP/1.1
Host: string
Authorization: string

The request does not use any URI parameters.

The request does not have a request body.

HTTP/1.1 200 OK
Content-type: text/html
<information on reporting errors>

Please contact coding.capability@abs.gov.au for other service support. Business hours are 9 am to 5 pm Monday to Friday.

These codes help identify issues on both the client and server sides, allowing for troubleshooting and resolution of HTTP request problems.

These are returned after synchronous or asynchronous coding operations:

The WoAG Occupation Coding Service and API have been security assessed by an independent registered assessor within the Australian Signals Directorate (ASD) Information Security Registered Assessors Program (IRAP) Program. This assessment found the Coding Service and API to have met the control and security objectives defined through the Australian Government Information Security Manual (ISM). 

Agencies may need to sign off in-house on using an external API, for business, legal, or security reasons. They may also need to check on their own behalf that the API response is from the address they sent the request to.

The following security controls, drawn from the ISM, are included to assist partner agencies in assessing their risks when using this service.

Please contact the ABS for support via email: coding.capability@abs.gov.au. We aim to respond to all enquiries as soon as possible. Our business hours are 9am to 5pm Monday to Friday.