In the past couple of days, I've been getting a server connection error pop up about two hours into any JupyterLab session I start on the RAP. Once I get the error for the first time, I usually find that I'm not able to restore the connection, and I am forced to terminate the JupyterLab session and lose any un-downloaded work. The full error message is:
Server Connection Error: A connection to the Jupyter server could not be established. JupyterLab will continue trying to reconnect. Check your network connection or Jupyter server configuration.
I've run into this issue when running both normal and high priority Spark cluster sessions on both mem1_hdd1_v2_x8 and mem1_hdd1_v2_x16 instances. My own WiFi connection doesn't seem to be the issue.
Has anyone else run into this and have any suggested workarounds? Thanks!
I usually lose connection from Jupyter in three occassions.
1) I was inactive for too long. Refresh browser, log in, and then it should work.
2) notebook timeout. You can see this in job log if you had indeed reached the limit.
3) losing worker due to spot market
If it you check and found that it's none of this, you might need to submit question to ukbiobank-support@dnanexus.com and share the project for the team to inspect more in detail.
Thanks! I've ruled out the other three options but memory issues seem like a possibility. Still new to the platform ? how would I be able to monitor memory usage as I work?
In the monitor tab, you can "view log" for the job. Then there is a tracking of resource usage for every 10 mins. This checking method would not always work if there is a sudden surge of memory usage though.
After checking all these aspect, if you still have issue, you may contact ukbiobank-support@dnanexus.com for them to look into your case.
I check with the support team and it seems that they have resolved it. One case is out-of-memory issue, while the other would need to change the configuration at the institution level.
Comments
8 comments
I usually lose connection from Jupyter in three occassions.
1) I was inactive for too long. Refresh browser, log in, and then it should work.
2) notebook timeout. You can see this in job log if you had indeed reached the limit.
3) losing worker due to spot market
If it you check and found that it's none of this, you might need to submit question to ukbiobank-support@dnanexus.com and share the project for the team to inspect more in detail.
I have a similar issue, getting "Server Connection Error A connection to the Jupyter server could not be established" on a regular basis.
Submitted a support ticket a week ago but so far haven't heard back from them.
I forgot to note the fourth possible reason when the worker is dead due to out of memory. This might be something you could watch out.
I my case it's none of the above and seems to be an issue with the SSL certificates.
Thanks! I've ruled out the other three options but memory issues seem like a possibility. Still new to the platform ? how would I be able to monitor memory usage as I work?
In the monitor tab, you can "view log" for the job. Then there is a tracking of resource usage for every 10 mins.
This checking method would not always work if there is a sudden surge of memory usage though.
After checking all these aspect, if you still have issue, you may contact ukbiobank-support@dnanexus.com for them to look into your case.
Thanks ? I checked and it looks like memory probably wasn't the issue either. I already reached out to support so I'll hope to hear back soon!
I check with the support team and it seems that they have resolved it. One case is out-of-memory issue, while the other would need to change the configuration at the institution level.
Please sign in to leave a comment.