Please, I am very new to the platform and still trying to understand a couple of things. How can I download demographic data and some biochemistry data for all participants from the RAP platform? the cohort browser only displays 30,000 items.
In my experience, you cannot download all the data from the RAP platform. You must compelet all your analysis in this platform, which is not free and prevent my further studies. The starting bill for free is only 40 pound, which is very short for a researcher from a lower income country or a student. Therefore, take care of your time and study plan.
Hi @Yong Chen? thank you so much for your responses. When you say all the data, does that mean that I cannot download the specific demographic characteristics (like sex, age, ethnicity) and a few biochemistry data for the >500,000 participants?
The first step is to get data of interest out of database into UKB-RAP project. You can do this using those two tools (table exporter or dx extract_dataset) I mentioned. Then you can analyze them using tools we provide or add your own tools. Or you can download them to your local computer if it doesn't violate the MTA that you signed with UKB. I couldn't speak for all MTA, but for what I saw, the pheno data can be downloaded. It doesn't mean that you can use that data yourself forever though. Please check your MTA for term of use and limitation.
The UKB-RAP is pretty cheap to use though since it's special pricing from AWS. When you download data, you would have to pay egress fee, so it's uncertain if it's actually cheaper if you download it. There is a risk in data leaking and compliance of system that you perform data analysis as well. If you can use code in common language like Python, bash, or R, it's probably less trouble to just use RAP.
Yong Chen makes a good point that it would take some time to learn since it's cloud computer, not the HPC or private computer. I personally find that this is useful skill to learn. Cloud computing is very popular in high budget industry and many biobanks are moving to this model since the data is too massive to move around. The up-coming webinar would have a big component on how to get data, so I'm sure it would be useful.
Hi {@005t000000149vjAAA}? , thank you so much for your great response and advice. I will definitely check my MTA to be sure of the term of use and limitations.
I tried using the table exporter after selecting a few variables just to test run the process in the cohort browser, only about 30,000 rows were exported into the RAP instead of the 502 376 participants. So, I was wondering if I did anything wrong in the process, or can't the table exporter export all the selected variables for all the participants? Regarding dx extract_dataset, I am not really sure how to go about it. Is there a step-by-step guide on using dx extract_dataset one can use to get the dataset into the UKB-RAP?
I only starting to learn how to use R, so I just have basic knowledge of the programming language. I would really love to improve my skills in that aspect and gain knowledge on how to run the analysis using cloud computing. What would you recommend as the best approach to doing this, particularly with respect to using the RAP for analysis? I am looking forward to the webinar on Thursday too.
Thank you so much for the grant link as well. This is so kind of you to share the information. I will look into it and apply.
Hi @Chinonso Odebeatu?, this is great timing, I just posted Query of the week #1 which deals with this very topic. Feel free to join me/Community in this discussion.
Hi @Chai Fungtammasan? thank you again for your reply. Please I have some questions:
I used the cohort browser to select the field of interest and saved it on the UKBRAP. I then opened the table exporter to convert the data into a CSV file. My question is does the table exporter capture all the 500,000+ participants in the dataset and not just the 30,000 rows shown in the cohort browser?
Secondly, after saving this file in the UKB RAP folder, how do I import it into RStudio for analysis?
Hi @Chinonso Odebeatu?, yes, the Table Exporter should export all the data from your cohort, not just 30k max. For Table Exporter, I normally define the columns to be exported (and I do not try to export everything). If you have already set up your UKB RAP account, I would suggesting running a test run.
Comments
11 comments
In my experience, you cannot download all the data from the RAP platform. You must compelet all your analysis in this platform, which is not free and prevent my further studies. The starting bill for free is only 40 pound, which is very short for a researcher from a lower income country or a student. Therefore, take care of your time and study plan.
It's probably best if you join the free live webinar training next Thursday.
https://hello.dnanexus.com/uk-biobank-research-analysis-platform-overview
Otherwise, you can check into the app Table Exporter or use dx extract_dataset to extract data you want.
Hi @Yong Chen? thank you so much for your responses. When you say all the data, does that mean that I cannot download the specific demographic characteristics (like sex, age, ethnicity) and a few biochemistry data for the >500,000 participants?
Thank you @Chai Fungtammasan? for your recommendation. I have already registered for the training. and will be attending it next week.
I have checked the recommended app previously and I'm yet to fully understand it.
The first step is to get data of interest out of database into UKB-RAP project. You can do this using those two tools (table exporter or dx extract_dataset) I mentioned. Then you can analyze them using tools we provide or add your own tools. Or you can download them to your local computer if it doesn't violate the MTA that you signed with UKB. I couldn't speak for all MTA, but for what I saw, the pheno data can be downloaded. It doesn't mean that you can use that data yourself forever though. Please check your MTA for term of use and limitation.
The UKB-RAP is pretty cheap to use though since it's special pricing from AWS. When you download data, you would have to pay egress fee, so it's uncertain if it's actually cheaper if you download it. There is a risk in data leaking and compliance of system that you perform data analysis as well. If you can use code in common language like Python, bash, or R, it's probably less trouble to just use RAP.
Yong Chen makes a good point that it would take some time to learn since it's cloud computer, not the HPC or private computer. I personally find that this is useful skill to learn. Cloud computing is very popular in high budget industry and many biobanks are moving to this model since the data is too massive to move around. The up-coming webinar would have a big component on how to get data, so I'm sure it would be useful.
For the grant, you can see detail here. https://community.dnanexus.com/s/question/0D5t000003rHbj3CAC/apply-now-uk-biobank-platform-credits-programme I can't remember the criteria for early career researcher, but I thought post-doc or new faculty is eligible. We will have an information session for this in April, but the applications has already opened and I heard that 50-70 grants has been given. You should apply.
Hi {@005t000000149vjAAA}? , thank you so much for your great response and advice. I will definitely check my MTA to be sure of the term of use and limitations.
I tried using the table exporter after selecting a few variables just to test run the process in the cohort browser, only about 30,000 rows were exported into the RAP instead of the 502 376 participants. So, I was wondering if I did anything wrong in the process, or can't the table exporter export all the selected variables for all the participants? Regarding dx extract_dataset, I am not really sure how to go about it. Is there a step-by-step guide on using dx extract_dataset one can use to get the dataset into the UKB-RAP?
I only starting to learn how to use R, so I just have basic knowledge of the programming language. I would really love to improve my skills in that aspect and gain knowledge on how to run the analysis using cloud computing. What would you recommend as the best approach to doing this, particularly with respect to using the RAP for analysis? I am looking forward to the webinar on Thursday too.
Thank you so much for the grant link as well. This is so kind of you to share the information. I will look into it and apply.
Hi @Chinonso Odebeatu?, this is great timing, I just posted Query of the week #1 which deals with this very topic. Feel free to join me/Community in this discussion.
https://community.dnanexus.com/s/question/0D5t000004SBm0eCAD/query-of-the-week-1
Hi @Ondreji?, thank you so much for your comments. Sure I will like to join the discussion
Hi @Chai Fungtammasan? thank you again for your reply. Please I have some questions:
I used the cohort browser to select the field of interest and saved it on the UKBRAP. I then opened the table exporter to convert the data into a CSV file. My question is does the table exporter capture all the 500,000+ participants in the dataset and not just the 30,000 rows shown in the cohort browser?
Secondly, after saving this file in the UKB RAP folder, how do I import it into RStudio for analysis?
Hi @Chinonso Odebeatu?, yes, the Table Exporter should export all the data from your cohort, not just 30k max. For Table Exporter, I normally define the columns to be exported (and I do not try to export everything). If you have already set up your UKB RAP account, I would suggesting running a test run.
For your Rstudio questions, see this doc page: https://dnanexus.gitbook.io/uk-biobank-rap/working-on-the-research-analysis-platform/using-rstudio-on-the-research-analysis-platform#working-with-data
Hi @Ondrej Klempir? Many thanks for your reply. This is very much appreciated
Please sign in to leave a comment.