DataLab

DataLab is the analysis solution for high-end users who want to undertake real time complex analysis of detailed microdata. Compare data services to see if detailed microdata in the DataLab is the right service for you.

Note: Government organisations seeking to leverage the DataLab cloud infrastructure to host their own data and manage their own end-users should consider the Secure Environment for Analysing Data (SEAD) service. 

• View and analyse unit record information
• Recent versions of analytical software, including R, SAS, Stata and Python
• Virtual access to files that remain in the secure ABS environment
• All analytical output that you want to use outside DataLab are checked by the ABS before release

Detailed survey and integrated microdata are available for approved projects, organisational users must be:

• government employees
• government contractors and individuals sponsored by government
• academics
• researchers from public policy research institutes
• sponsored by government

All users need to also meet ABS safe people criteria, including researchers who:

• belong to an Australian organisation (international researchers and organisations will be considered on a case by case basis)
• belong to an organisation with a Responsible Officer Undertaking (ROU) in place with the ABS
• are located in Australia when accessing the microdata
• have completed all relevant undertakings and declarations
• have the ability to use at least one of the statistical analytical languages available in the DataLab
• have at least three years of either quantitative research experience or university study with a significant component working with quantitative data, or have a referral from an experienced researcher working on the same project
• have an approved safe project that is for statistical and/or research purposes and demonstrates public value
• have completed ABS safe researcher training and refresher training as per ABS refresher policy
• meet additional criteria that apply to specific microdata.

The ABS is committed to keeping the ABS DataLab safe and secure. We have a strong data protection culture and extensive experience in keeping data secure as Australia’s national statistical organisation and as an Accredited Data Service Provider. The ABS DataLab is hosted in Microsoft Azure and meets PROTECTED level security standards as prescribed in the Australian Government Information Security Manual (ISM). It is subject to Independent Security Registered Assessors Program (IRAP) certification, ongoing security audits and robust IT security testing and patching delivering the Safe Settings aspect of the Five Safes Framework.

The technology underpinning the ABS DataLab includes:

• data encryption at rest to mitigate against unauthorised access to microdata
• Azure Storage Accounts to securely hold individual research products and allow querying from authorised users
• cloud servers (including backup servers) hosted exclusively onshore, with access only authorised for use in Australia unless approved by the ABS
• closed network virtual machines to provide secure, isolated research spaces for the analysis of microdata
• guarded access through multi-factor authentication and workspace segmentation inhibiting data sharing between projects
• a DataLab Product Storage Account protected with Microsoft Defender providing threat detection against malicious/unusual behaviour.

The ABS employs the above with a focus on industry standard security posture management to provide a safe and secure platform for policy and program delivery work.

• Designed specifically for use within the DataLab environment
• Direct identifiers (such as names and addresses) removed
• Further appropriate confidentiality applied within the context of the other security features of the DataLab
• Topics include Census, health, education, labour force, Aboriginal and Torres Strait Islander peoples, migrants, crime, business, disabilities, ageing and carers
• Datasets include ABS survey results, administrative data collected by other organisations and integrated datasets
• Data item lists are linked in detailed microdata topics in the DataLab

Approved users can access standard detailed microdata in DataLab for approved projects. This includes:

• ABS survey and census collections
• data ABS has collected from other organisations (with custodian approval)
• integrated microdata such as:
  • Person Level Integrated Data Asset (PLIDA)
  • Business Longitudinal Analysis Data Environment (BLADE) Core plus BLADE standard module (various ABS surveys), Intellectual Property Longitudinal Research Data (IPLORD) and Merchandise Imports and Exports
  • PLIDA/BLADE linked data

Additional charges apply for customised data integration services.  

DataLab access incurs an annual charge. This charge is based on the number of analysts with virtual machine access in a project. The charge covers the annual costs of:  

• project establishment and ongoing administration and support 
• researcher onboarding (including training and refresher training) 
• changes to analysts and discussants within a project 
• standard virtual machine access for analysts 
• standard software access for analysts (e.g., R, Python, Stata) 
• project storage up to one terabyte 
• standard output and input clearance 

The ABS is committed to supporting the DataLab service and subsidising DataLab users. In 2023-24, increasing costs and budget constraints require the ABS to move towards a more sustainable partial cost recovery arrangement with all our clients. The below updated charges will allow the ABS to maintain its service levels as well as deliver critical system and infrastructure enhancements.   

The ABS is adopting a staggered approach to price increases to minimise the impacts on existing projects. Existing projects are projects established before 1 July 2023. Existing and new projects will be subject to different pricing models for 2023-24 and 2024-25. Please see the below for more details.  

Please note: 

• Individual quotes will be prepared for projects with over 25 analysts. 
• An increase in the number of analysts to the next tier will incur an additional charge equivalent to the next tier.
• Significant changes in project scope may result in the establishment of a new project.
• Project extensions after 1 July 2023 will be charged at the new project rates.
• Access to non-standard services are subject to additional charges, please see ‘Additional non-standard access and services charges’ table below.
• All charges are calculated quarterly, based on the month of request. For example, a project commencing in October will be charged the annual fee and any non-standard DataLab access based on three quarters of the financial year.

Existing projects are projects established before 1 July 2023.  

Tier 1 and 2 projects (projects with under 10 analysts) 

• Tier 1 and 2 projects will continue to be charged the same annual rates for financial years 2023-24 and 2024-25.
• Prices will increase from 2025-26 for all Tier 1 and 2 projects.

Tier 3 projects (projects with 11 to 25 analysts) 

• Tier 3 projects will be charged half the 2022-23 annual charge of $10,000 ($5,000 excluding GST) for 1 July 2023 to 31 December 2023, with prices increasing from 1 January 2024.
• From 1 January 2024, Tier 3 projects will be subject to a new charge rate. 
• Tier 3 projects will be charged half the new 2023-24 annual charge of $30,000 ($15,000 excluding GST) for 1 January 2024 to 30 June 2024. 
• The delayed roll out of the new charging model provides project leads the opportunity to decrease usage or close their project before prices increase. 

Tier 4 projects (projects with over 25 analysts) 

• Tier 4 projects will be charged custom pricing from 1 July 2023 under the new pricing model.

New projects are projects established after 1 July 2023.  

The annual charge for projects with up to 10 DataLab analysts is covered under the ABS/Universities Australia agreement. These projects will not be subject to an annual charge for the duration of the current agreement, in place until 31 December 2023.

Non-standard charges apply to all projects for services that are not within the scope of the annual charge. Non-standard charges include access to SAS, Databricks, non-standard virtual machines and storage above one terabyte. Non-standard charges also apply to increased service levels, such as priority clearance and high service level. More information on non-standard access and services are in the tables below. 

For criteria, refer to Who can access the DataLab

Organisation approval

• Your organisation must have a verified Responsible Officer Undertaking (ROU) in place with the ABS. If one does not exist, your organisation will not be available for selection in the myDATA beta online project proposal.
• To check if there is an active ROU for your organisation, go to the myDATA user portal homepage and select Dashboard/Organisation.

Researcher approval

• You must have a commitment to protect the confidentiality of data.
• Every member of your project team who will see or discuss uncleared outputs (whether or not they will be using DataLab) needs to be approved.

Project approval

• Projects must be for statistical and/or research purposes and provide public benefit.
• Projects must not be for compliance or regulatory purposes.
• Every project needs to be approved by the ABS.
• Projects for or about Aboriginal and/or Torres Strait Islander peoples may be subject to a Cultural Review by the Centre of Aboriginal and Torres Strait Islander Statistics at the ABS.
• Some projects also require consideration and approval by data custodians.

Refer to What is DataLab and Using DataLab responsibly for more information.

Registration of an account will allow you to:

• create a project proposal within the myDATA user portal
• enrol in DataLab safe researcher training
• collect forms for your onboarding process after training has been completed
• draft and review projects you are participating in.

Register in the myDATA Beta Portal and agree to the Conditions of use

• Use your organisation email address. If you are a user in more than one organisation, you will need to register separately using the email address for each organisation.
• Authenticate your account - myDATA will automatically email your registered account with steps to authenticate.
• If you encounter errors in myDATA, please submit a System support query.

You can complete your training while your project proposal is underway.

Access will not be granted until DataLab safe onboarding is completed. For the onboarding process, the following documents are required to be submitted:

• Individual Undertaking and DataLab Declaration of Compliance
• DataLab quiz
• additional documents relevant to particular datasets, if requested by ABS
• if you are not an experienced researcher, you may need to submit a DataLab Referral, please email data.services@abs.gov.au to request this form.

For further information, please refer to the myDATA beta user guides section below.

DataLab is enabled by cloud infrastructure, which may be blocked by some organisations’ firewall settings. 

ABS cannot make changes to external organisations' infrastructure. Project Leads need to supply the information below to each organisation participating on this project. 

Network/IT Security sections in each organisation need to review and make changes to authenticate access. This only needs to be done once per organisation.

There are four steps which need to be applied to each organisation’s security settings before the project start date to enable access to DataLab.

1. Enable authentication to the tenant

Users need to authenticate to one of ABS Azure Active tenants, which may be strictly controlled by government agencies and academic workplaces. Authentication must be enabled to the tenants:

• mydata.abs.gov.au
• absmydata.onmicrosoft.com

2. Allow user access to URLs

Users will need to access the following URLs:

• DataLab production portal: datalab.abs.gov.au and gw.datalab.abs.gov.au
• Citrix portal: absdatalab.cloud.com

3. 2020 version of Citrix Workspace client installed

The originating client machine must have a recent version of the Citrix Workspace client installed. Here is a link to the Citrix Workspace download page.

4. Enable HTTPS connections

All Remote Desktop client connections to ABS DataLab go via Citrix Cloud service. Your organisations Network/Security area will need to enable HTTPS connections to the following:

• *.citrix.com
• *.cloud.com
• *.*.nssvc.net

Organisations that can't enable all subdomains can whitelist using wildcards to prevent future connectivity issues to Citrix. For more information refer to Citrix Product Documentation.

Customers who can’t enable all subdomains can use the following addresses instead:

• *.g.nssvc.net
• *.c.nssvc.net

SSL/TLS inspection must be bypassed for *.nssvc.net as it can break connections to Citrix Gateway Service.

1. Enable authentication to the tenant

Users need to authenticate to one of ABS Azure Active tenants, which may be strictly controlled by government agencies and academic workplaces. Authentication must be enabled to the tenant:

• absmydata.onmicrosoft.com

This tenant is located in Azure Australia East and Azure Australia Central regions. 

2. Allow user access to URLs

Users will need to access the following URLs:

• DataLab production portal: datalab.abs.gov.au and sead.abs.gov.au

3. Configure your organisation's network to allow outbound connections to the following addresses required for Azure Virtual Desktop (AVD):

• login.microsoftonline.com 
• *.wvd.microsoft.com 
• *.servicebus.windows.net 
• go.microsoft.com 
• aka.ms 
• learn.microsoft.com 
• privacy.microsoft.com 
• query.prod.cms.rt.microsoft.com 

These addresses all utilise the TCP protocol and outbound port 443 for communication. 

Contact data.services@abs.gov.au for further assistance.

myDATA beta is a system to manage DataLab projects. myDATA beta will be released in 4 stages:

• Beta 1: Training and onboarding (released)
• Beta 2: New project proposals and ABS feedback (released)
• Beta 3: Project change requests (pending)
• Beta 4: Approvals and project closure (pending)

With the release of Beta 2, you will be able to create, submit and edit your project proposal via the myDATA user portal.

The myDATA Beta user guides are below and can be downloaded as PDF files.

1. Register and activate user account (download pdf)
2. User portal dashboard navigation (download pdf)
3. Self-training enrolment (download pdf)
4. Download and return forms (download pdf)
5. Create and submit a project proposal (download pdf)

The ABS privacy policy and DataLab privacy notice outline how the ABS handles any personal information that you provide to us.