Azure Storage Explorer

The default method of loading data to SEAD is to use Microsoft Azure Storage Explorer (ASE), which will need to be setup and managed by your organisation’s IT department. Refer to Technical specifications for requirements on network access for Azure File storage using ASE. Ensure that when you are arranging access to ASE that you have access to V1.36 or later available. To open ASE, search for the application in the start menu, then select open.

NOTE: If users/analysts are using Azure Data Lake containers with Databricks, ASE is available inside users’ virtual machines (VM) as an alternative to AzCopy to manage and transfer their files between their file share drives (output, project, etc) and blob storage. Users can refer to the 'Azure Storage Explorer User Guide' in the shared library drive for more information. Users do not have access to upload or download data via ASE outside of their VM. Data ingress and egress to SEAD is managed by data administrators only.

Some organisations may require Data administrators to configure a proxy for web connections. Relevant organisation IT support may need to provide the following details to administrators to configure their proxy settings in ASE:

• A proxy address and port to allow ASE to reach the internet.
• If the proxy requires authentication, the credentials that should be used to log in.

If unsure, contact your relevant organisation IT support to assist. 

To configure proxy settings in ASE, from the options menu select 'Edit' and 'Configure Proxy'.

From the proxy settings, depending on your organisation’s requirements you will need to select the ‘Source’, ‘URL’, ‘Port’, and if required, a username and password. Then select 'Ok'.

If you are having issues authenticating or accessing the storage accounts after configuring the proxy settings, or the proxy is returning to default you can try to configure the proxy through the settings menu, ensuring that the 'Auto manage proxy settings' option is disabled.

To access proxy configuration from the settings menu, select the ‘Settings’ cog and locate the proxy details from the application settings:

Within the application, select the 'Account Management' view (circled red) on the menu bar and select ‘Add an account’.

A pop-up window will appear, on the ‘Select Resource’ page, select ‘Subscription’.

On the following ‘Select Azure Environment’ page, select ‘Azure’ then select ‘Next’.

The system will redirect to a browser window.

Enter the SEAD credentials for your Data administrator account, or select the account name from the list if you have recently logged in.

A new window will open stating the account has been authenticated. If you don’t have the Data administrator permissions, you will receive an authentication failure. Your account will be added to ASE and you can access the SEADpods storage accounts.

If you are unable to see any of the files/file containers after logging in, your session may have timed out and you will need to re-authenticate. This is indicated by the notification banner displayed in Fig. 1. Below.

To resolve this, select ‘Manage Accounts’ from the reauthentication notification at the top of your screen or navigating to the 'Account Management' page (person icon) from the left navigation panel in ASE. Once on your profile select 'Reauthenticate now' (circled red) beneath the account you are re-authenticating.

A new window will open in your web browser. Follow the prompts to sign in using your Data administrator account.

Once authenticated, the browser will display the following message:

Note: In some cases, you may receive a notification stating that the page cannot be reached. If this occurs, close the window and return to ASE where your account will be active. If it is not active, refresh your view.

Note: It is the responsibility of the SEADpod Project Owner to ensure compliance with the Project Owner’s legal requirements, rules and obligations pertaining to data input and outputs.

Storage Accounts are accessible under the 'DataLab-Prod' subscriptions to authenticated administrators. There may be multiple DataLab subscriptions visible to SEAD Administrators in ASE, each containing different storage accounts. Storage accounts are spread across multiple subscriptions because of load-balancing mechanisms in place to manage Azure limitations. SEAD administrators do not have permission to move projects to specific subscriptions.

There are five types of 'Storage Accounts' in ASE:

• Gateway storage account, prefixed with ‘gateway’. There will be one gateway storage account per SEADpod used to upload data via folder name and destination codes (refer to gateway storage accounts below).
• Product storage account, identified by the storage accounts starting 'prdct'. There will only be one Product storage account per SEADpod. This storage account lists all available products in the SEADpod. Users are not able to write to this drive in their VM.
• Project storage, identified in Azure starting with the ProjectID. There will be one storage account in ASE per project created in the SEAD portal. The project storage account allows access to the Projects and Outputs Drive for that project.
• Pod Library drive, is accessible to SEAD data administrators. The contents of this storage account is accessible to all users within the SEADpod.
• Library drive, which is only accessible for loading to by ABS administrators.

Gateway storage accounts scan all ingressed data to block malicious files. Data administrators upload through the gateway, and safe files are automatically transferred to the destination account. Malicious files remain quarantined. If this happens contact your ABS system administrators at sead.support@abs.gov.au.

Note: Any files exceeding 50GB will not be scanned and will remain in the gateway container for 24 hours before being deleted.

Each SEADpod has a single gateway storage account. Within the gateway, all file shares and datalake containers for the SEADpod appear as individual folders. Each folder is labelled with one of the following destination codes that identifies whether it is a file share, or datalake container:

•  r = product fileshare folder
• a = product “datalake” Azure data lake storage container
• p = project "project" fileshare folder
• o = project "output" fileshare folder
• d = project "datalake" Azure data lake storage container
• x = pod library fileshare folder

To quickly find the correct storage account in ASE, use the “Manage Files” button in the SEAD admin portal.

On the projects interface, select the project you want to upload or manage files for and select the ‘Manage Files’ button. 

This will open the project’s 'ASE manager menu' which provides accessible shortcuts to the project, output, and data lake container's gateway folder or file shares.

On the products interface select the ‘Actions’ button for the product you are wanting to upload or manage files for and select ‘Manage Files’.

This will open the products ASE manager menu which provides accessible shortcuts to the product's file share and data lake container.

The gateway container can be accessed for the specified product or project by selecting the ‘Upload’ shortcut for the location you want to upload data to. The file shares or blob container can also be accessed directly by selecting the appropriate ‘Read/Write’ shortcut.

To access the pod library, you must navigate through ASE directly as there are currently no shortcuts available through the SEAD portal. The gateway container for the pod library is labelled as 'x-podlibrary' (circled red in the image below). The 'podlibrary' file share is located in the ‘prdct’ storage container (circled blue).

Before uploading any data you must first create a product shell (refer to ‘Creating Products’) or a project shell (refer to 'Creating new projects’) from the SEAD administrator interface. This will automatically enable an Access Control List (ACL) in ASE. ACL's define who can access a resource and the actions they are able to perform. To see the newly created shell in ASE, data administrators will have to log out and back in again for access control lists to refresh. Product and project folders created directly in ASE are not configured with ACL protection and will not be accessible to end users.

After creating your shell, ensure your Azure account is activated (refer to Setting up Azure Storage Explorer) and you have the data administrator role. 

Using the 'Manage Files' button found on the ‘Products’ or ‘Projects’ interface within the SEAD portal, select the desired location’s (file share or data lake container) ‘Upload’ button. 

A pop-up window will appear prompting ASE to open. Select 'Open'. If the pop-up menu does not appear after clicking the shortcut link, check that pop-ups are enabled in your browser settings and are not blocking the pop-up.

Note: It is recommended that you open ASE and authenticate your account before using the shortcuts in the SEAD admin portal. If ASE does not launch after clicking the shortcut, check that you have authenticated to ASE and try again.

When ASE opens you will be asked to choose how you would like to open the direct link. Select ‘Navigate to Container’.

You will then be directed to the selected gateway folder or file share container.

If accessing the gateway container directly through ASE, open the ‘gateway’ storage account, then open the ‘Blob Containers’ file share. You will need to locate the Product or Project’s file share in the folder list. Each folder name includes the project or product name along with the destination code. For details on the destination codes, refer to gateway storage accounts.

Tip: Right-click on a storage account and select 'Pin to Quick Access' to allow you to quickly locate the storage account instead of searching through the list of storage accounts.

On the gateway storage account you can select 'view all' which opens the full list of gateway folders and corresponding destination codes.

To locate the destination folder, start by inputting the short name in the bar (recommended if the SEADpod has a large number of folders), or scrolling through the list until you find the desired folder.

To add data, either drag and drop the files into the intended gateway folder, or select the ‘Upload’ button, ensuring you have the correct gateway folder selected. Select either 'Upload Folder' or 'Upload Files'.

On the following window, select the three dots to locate the required folders or files.

Select the product folder or file name and press ‘Select Folder’.

On the following window, check the upload paths are correct and at the appropriate file/location level. If correct, select ‘Upload’.

Note: Only data administrators within your SEADpod can view objects in ASE. ASE denies authentication to administrators who do not have the data administrator role within the SEADpod.

After files have been successfully scanned, at the bottom of the screen in ASE, the ‘Activities’ box will indicate if the upload has been successful. If the upload has failed, retry the above steps.

Files that have been uploaded through the gateway container will automatically be transferred to the destination storage account once they have been confirmed safe. Data administrators can check that this has been successful, under the ‘gateway’ storage account open the ‘Tables’ folder and open the ‘GatewayLogs’ where the upload status of your files will be shown.

You are also able to use the SEAD portal to navigate to the ‘Gateway Logs’ by selecting the 'Manage Files' button, on either the Product or Project page which contains a shortcut to link to the Gateway Logs table, called ‘logs table’.

Next, quality check the file upload by ensuring the transfer of the files from the gateway to the destination storage account on the gateway logs table. The gateway logs table contains information on the file upload, including the partition key (folder name and destination code), destination container, file name, information, status (complete, failed, in progress), and time stamp.

To view the data that is currently available to your end users, or to download/egress data from SEAD, you will need to navigate to the file shares directly, rather than the gateway storage account.

From the “Manage Files” button in the SEAD portal, select the desired location’s (file share or data lake container) ‘Read/Write’ shortcut. This will direct you to the selected file share or data lake container where you will be able to see the files currently available to your end users.

This will prompt ASE to open. A pop-up window will appear, select 'Open'. If the pop-up does not appear after clicking the shortcut link it may be blocked. Check in your browser settings that pop-ups have been enabled.

When ASE opens you will be asked to choose how you would like to open the direct link. Select ‘Navigate to Container’.

From the file share you can egress the files out of ASE. Use the download function (beside the upload button) in ASE to egress data  to your organisations local data store.

NOTES:

• SEAD users/researchers do not have data administrator permissions to ingress or egress data from SEAD. Users have read and write access to files made available to them in the 'My Products' folder, and the Project/Output folders linked to their workspace, so they can delete files in these folders.
• If the folder in the ASE directory is empty, the files meant for upload may not have been loaded. Attempt to check through ASE, ensure files have actual content (size should not be 0 KB). Also be aware that the size of files uploaded to SEAD are smaller, due to compression. For some files, the CONTENT-MD5 may be blank. This is because CONTENT-MD5 is specifically related to Azure Blob Storage and since the storage in SEAD is Azure Files/file share based, it is unused/blank.
• Blob containers are accessible in ASE as an alternative for file shares, primarily used with Azure Databricks enabled VM's.

Data administrators can restore files that have been backed up within the last 14 days. To do this, navigate to the project share for the relevant project and click on the drop down ‘Current’ or ‘View Share Snapshots’ button. 

Note: the Data Lake and Databricks managed storage are not backed up due to implementation restrictions. We recommend advising users to regularly back up their Databricks storage to their Project (P:) drive.

Here you can view backup snapshots for the last 14 days, the timestamps are in UTC time zone which equates to 5AM the following day AEST. E.g. 2023-05-01T18:47:05.0000000Z equates to 2023-05-02 4:47 AM AEST.

If the file you are restoring still exists in the ‘Current’ snapshot and you don’t want to overwrite it from the restore point, create a backup folder and copy the file into it.

Select the time of the backup to restore, right click the file and select ‘Restore Snapshot’.

Confirm the restore by selecting ‘Yes’ in the pop-up window. 

Note: If you haven’t renamed or moved the current version of the file into a folder it will be overwritten.

Verify the restore completed successfully in the activity log. 

Each SEADpod will have access to their own ‘Pod Library’ managed by Data administrators through ASE. All files that are added to the Pod Library will be accessible to all end users in the SEADpod, regardless of the project they are on, as an additional network drive (Z:). Users will have read only access to the Pod Library drive in their VM.

The Pod Library is separate from the existing Library drive (L:) which is shared across all SEADpods and is available to all users. The Library drive will continue to be managed by the ABS and will continue to host system documentation and usage instructions.

Unlike the projects and products, there are no shortcuts to ASE from within the SEAD portal. To locate the Pod library drive in ASE, the file shares will be located under the product (‘prdct’) storage account. From the file share you will be able to view or delete the contents of the Pod Library. To add files to the Pod Library, in the gateway container the Pod Library destination code and folder name is x-podlibrary.

The contents of the Pod Library are at the discretion of the SEADpod and your requirements around files being accessible to all projects. Keep in mind the ABS guidance around safe access and management of code, packages, and software being uploaded to SEAD. Refer to code and package load for more information.

Check that you have ticked all subscription boxes against your login credentials in 'Account Management' settings. You may also need to re-authenticate your ASE credentials, or confirm your proxy settings with your local organisation’s IT administrator. 

Check that you have ticked all subscription boxes against your login credentials in 'Account Management' settings. You may also need to re-authenticate your ASE credentials, or confirm your proxy settings with your local organisations IT administrator.

Check the 'Gateway Logs' to verify the transfer status and location of your file.

An ABS system administrator will contact you. If the file is larger than 50GB, you will need to chunk it up before sending it through the gateway again. It is the responsibility of the file owner to ensure that files ingressed are free of malware.