How to update dispensed data

Data Updates

UK Biobank (UKB) periodically releases new data and makes updates to existing data on the UK Biobank Research Analysis Platform (UKB-RAP). Whenever the UKB-RAP updates its copy of UKB data, it will be indicated by a new data release version. To get up-to-date data in your project, you can check if an update is available and perform a data refresh to update the dispensed data. The data update process will synchronize the project against the latest data release. This affects both tabular data and bulk file data.

Tabular data

For tabular data, the update process will make in-place changes to the previously-dispensed database (Type/Class = Database), including any schema changes (such as for new fields) and row updates. A new associated dataset (Type/Class = Dataset Record) will be generated, and researchers should use this new Dataset Record item. Whilst previous cohorts and dashboards will persist in the updated project, they will not work properly. To migrate old cohorts or dashboards to the new dataset please run the Rebase Cohorts And Dashboards app.

Bulk file data

For bulk file data, the update process will dispense any new files, potentially rearranging previous folders if folder names have changed.

Separate current data from new data

If you would prefer to have the latest version of the data in a separate area, you can also dispense a new project which will avoid updating a project in-place. When both projects exist, it will be possible to copy any derived variables or other researcher files from the old project to the new project using the dx copy command from the dx toolkit. The dx toolkit is preinstalled in both the JupyterLab terminal and the ttyd app. For more on dx commands, please see the following resources:

How do I check for data updates in my project?

To check for updates, go to the "Settings" page of a dispensed project and click the "Check for Updates" button in the "UK Biobank" section.

If the dispensed data is already up-to-date, you should see the below:

If there is a data update available for the dispensed data, you should see the below:

When data updates are available, you can click the "Show Update" button to see more information about the latest data update.

NOTE: We highly recommend not to launch a large number of jobs and perform clone/copy operations on a large number of objects in the requested project.

By clicking the "Start Update" button, you kick off the data refresh process. You can check the progress of the update process from the "status" section.

After the refresh is done, the status will return to "Ready".

Why is my data refresh taking a long time?

Please be aware that if you decide to update the data in your UKB project during a particularly busy time (just following a new data release), high traffic can lead to particularly long wait times. If it takes longer than a week for your new project or your refresh to load, it is recommended that you contact DNAnexus support (ukbiobank-support@dnanexus.com).

Why am I receiving an error message?

When clicking the "check for updates" button, you may receive an error message like the following:

This particular error message can occur if you were not previously on the most recent version of the data (for example, you can only refresh to v19 if you were previously on v18, rather than v17). If this happens to you, make sure to create a new project before deleting your current one. Then, copy across any files you need to your new project using the dx copy command in the dx toolkit.

Note: If your AMS application is currently legacy Tier-0, or if it was Tier-0 at the time of your most recent dispense, then the project will not be eligible for updates. Please dispense a new project.

If you need additional guidance or support please submit a ticket

Related to