How to submit a large input-json with dx run?

• 

  Chai Fungtammasan DNAnexus Team

  • 06 October 2022 15:45

  There is a limit in term of how long each API call can be and how many input files are used for each job. Are you looking for an operation that would need all files together at once or you would need to process all of them and plan to use this job to orchestrate it?

   

  Assuming the former, you might need to create an applet that take folder path or a file containing list of input file as input and internally consume the file within applet itself.

   

  Assuming the later, you may find this guideline useful. https://dnanexus.gitbook.io/uk-biobank-rap/science-corner/guide-to-analyzing-large-sample-sets#create-job-submissions-where-each-job-processes-100-samples

  This is the simplest possible way to manage it, but you can use more advance trick like using each instance for more than one input file as well, but it's harder to optimize.

   

  0
• 

  Former User of DNAx Community_18

  • 06 October 2022 16:06

  Kind of both. I'm trying to do a scatter-gather with wdl (here is the code). So each file is first processed independently (so 60K jobs) and then those outputs are processed together in one job.

   

  Thanks for the links - I will take a look.

   

  0

Please sign in to leave a comment.