Spark Applet fails because of dx-spark-submit

Ba Khuong Dang

Hello,

I am trying to run a SparkApplet follwing this document (https://github.com/UK-Biobank/UKB-RAP-Notebooks-Access/blob/main/Example-applets/spark-example-applet/), and I am using `dx-spark-submit` high level command as suggested in (https://documentation.dnanexus.com/developer/apps/developing-spark-apps/dx-spark-submit-utility). But my applet keeps failing because of error: “dx-spark-submit: command not found”. Does anyone know how to fix this error, or it is not supported in `dx-toolkit` anymore? 

Similarly, I try to run my analysis with JupyterLab SparkCluster but it always disconnect after a certain time. I tried to run JupyterLab from CLI (according to DNANexus documentation it will keep me login), but I still have the issue of logging out after inactive period. I think this is the problem of UKBiobank timeout session rather than DNANexus platform. How do you run your analysis (Pyspark script) when it takes long time? 

Thank you very much, and I appreciate any documentation or examples that I can look into. 

Comments

0 comments

Please sign in to leave a comment.