Why is data missing from my UKB-RAP project?

There can be various reasons for this:

• The data dispensed will depend on the Access Management System (AMS) Application Tier of the project

Cost Tier 3 data, such as the Whole Genome Sequencing (WGS), will not appear in projects that are Tier 1 or Tier 2.   Check the Tier of the project in AMS and the Cost Tier of the missing data in the Showcase page for the field. If you have an older style (Tier 0) application and need additional data to be available in your UK Biobank Research Analysis Platform (UKB-RAP) project, please submit a ticket about upgrading the Tier.

• Data is dispensed in stages (bundles) and the initial dispense does not include the WGS data

To dispense more data bundles, a project Administrator should go to the Settings page and click "Dispense more data".  Projects with all three bundles dispensed will be approximately 26 PiB.

• Many fields in the UK Biobank database are only available for a subset of participants

Many UK Biobank fields only have data for a subset of participants. Some of these fields will never have full data. Some of these fields are updated on an ongoing basis.   The field page in Showcase will provide an indication of the maximum number of UKB participants that might have data for the field at the present time, but see the next point.  

• Data may be listed on Showcase but not yet in the main UKB-RAP copy

New data will be listed in Showcase for several weeks before it becomes available in the main UKB-RAP copy.  Compare the Showcase field version date with the latest UKB-RAP version date.   If the Showcase field version date is later than the latest UKB-RAP version date, then the most recent update for that field is not yet available.  It will become available in the UKB-RAP main copy at the next UKB-RAP update, listed in "upcoming data releases" in the data releases page.  

• Data may be in the main UKB-RAP copy but not in your dispensed UKB-RAP project copy

Compare the latest UKB-RAP version date with the project dispense date. The project dispense date is part of the filename for the Dataset Record. If necessary, update your project or dispense a new project to receive the newest data. 

• Data may be present in your dispensed UKB-RAP project copy, but not available in the Cohort Browser when opened with an old Dataset Record item

Projects that have been updated will have multiple Dataset Record items.   To work with the latest data in the Cohort Browser it is necessary to open the Cohort Browser using the latest Dataset Record item.   Using the Table Exporter will automatically use the latest Dataset Record item.   When using Spark commands to access the data in the Parquet database it is necessary to reference the latest Dataset Record item.

• Tabular data and Bulk data are stored differently

Check whether the field of the missing data is listed in Showcase as Bulk or as Data (which means Tabular Data), and search in the Bulk folder for Bulk fields or the Parquet Database for Tabular Data.   For more details see the finding data article.  The Cohort Browser only works with the data in the Parquet database.  

• A small amount of data is being removed from the UK Biobank dataset

UK Biobank participants can withdraw consent to the use of their data. These withdrawn participants are frequently removed from the main UKB-RAP copy of the data. Individual-level Bulk files for withdrawn participants are periodically removed from dispensed UKB-RAP projects.   Researchers should remove the data of withdrawn participants from the Parquet database and from population-level Bulk files in each dispensed project  whenever they receive an email about withdrawals, and whenever the project shows that an auto-removal is pending. They may remove withdrawn participants more frequently.   For more details see auto-updating withdrawals and the withdrawals list article.

• If the dispense process did not complete successfully, the data may be corrupt or some might be missing   

This is unlikely, but not impossible. If you have checked all the other possibilities described in this article, you could consider dispensing a whole new UKB-RAP project .

• Data might be Restricted  

Some fields are Restricted and require special authorisation. Some fields are Restricted and not available to any researchers at present. Check whether the data field is marked in Showcase as Restricted.

• Bulk data might be filtered with a file limit in the UKB-RAP viewer 

Only the most recently updated 1000 files are displayed in each sub-folder within the Bulk folder when browsing through the UKB-RAP. The rest of the files are present in the project but not obvious in the folder view.  The total number of items in a folder should be displayed at the bottom of the page - for an example, please see the screenshot below.

To see a different set of files, you can use the sort and filter options in the viewer. Alternatively, you may be interested in using the dx-toolkit to navigate through files using the Command Line Interface (CLI).

More specifically, you can use the dx find data command to search for a field or dx ls to list all objects in a folder. For more information about the CLI, see the DNAnexus video tutorials. 

Total number of files example:

If you need additional guidance or support please submit a ticket