I would like a way to have interactive access on DNA Nexus, such as by running ttyd, while I also have many jobs (>100) currently running and more queued.
I have tried to submit a new ttyd job, with priority either "normal" or "high", e.g.:
dx run ttyd --ssh --instance-type mem1_ssd1_v2_x4
However, this job never starts, even though my previously submitted (normal priority) jobs continue completing and new ones starting.
Is there a way to get such a job to start?
I am wondering for which use cases you would like to run one more high priority ttyd job? Would it be just for interacting with dnax project? Monitoring jobs, checking output files, manipulating data in folders? If so - and if you do not need to perform additional high priority computation, the dxtoolkit installed locally could help.
I would like to test or debug computational pipelines / code in development that may need access to Gb of data, including genotypes, in which case I cannot download the data to operate on locally.
The unfortunate thing is that when I am also running many other jobs, I'm then unable to get a ttyd job to run at all, regardless of the priority assigned to it. (Is this normal? Shouldn't a high priority job take precedence over existing normal priority jobs that are already queued?)
My understanding is that "High priority" is more about highly stable machine that cannot be interrupted by spot. And the sentence "need to be executed as quickly as possible" would be more about starting job as soon as possible (and only if you are not reaching the job limit), and not much about moving the high priority job to the top/head of (priority) queue.
Maybe someone else here at Community can help. However, this seems to me as an advanced production related question, so I would ask directly at ukbiobank-support@dnanexus.com.
I checked with ukbiobank-support@dnanexus.com and found that API for this does exist. However, the feature seems to be available to only for licensing customers.
Comments
4 comments
I am wondering for which use cases you would like to run one more high priority ttyd job? Would it be just for interacting with dnax project? Monitoring jobs, checking output files, manipulating data in folders? If so - and if you do not need to perform additional high priority computation, the dxtoolkit installed locally could help.
https://documentation.dnanexus.com/downloads
Thanks Ondrej.
I would like to test or debug computational pipelines / code in development that may need access to Gb of data, including genotypes, in which case I cannot download the data to operate on locally.
The unfortunate thing is that when I am also running many other jobs, I'm then unable to get a ttyd job to run at all, regardless of the priority assigned to it. (Is this normal? Shouldn't a high priority job take precedence over existing normal priority jobs that are already queued?)
I see ? . I think that what you described would be intended behavior. I am getting information from here: https://dnanexus.gitbook.io/uk-biobank-rap/working-on-the-research-analysis-platform/managing-job-priority#priority-levels
My understanding is that "High priority" is more about highly stable machine that cannot be interrupted by spot. And the sentence "need to be executed as quickly as possible" would be more about starting job as soon as possible (and only if you are not reaching the job limit), and not much about moving the high priority job to the top/head of (priority) queue.
Maybe someone else here at Community can help. However, this seems to me as an advanced production related question, so I would ask directly at ukbiobank-support@dnanexus.com.
I checked with ukbiobank-support@dnanexus.com and found that API for this does exist. However, the feature seems to be available to only for licensing customers.
https://documentation.dnanexus.com/developer/api/system-methods#api-method-system-bulkrankupdate
https://documentation.dnanexus.com/developer/api/running-analyses/applets-and-entry-points#api-method-job-xxxx-update
Please sign in to leave a comment.