How do I navigate to the log file in Linux given on a HPC cluster (with many users)? I tried to do search, but there are many users so not helpful to try to find log file (gives many access denied messages). When I can access the log file then I can figure out if it is a file issue or a memory issue (given resource allocation), or something else, but right now I do not know.
Version of Qiime2-2018.6 in conda environment on HPC cluster account (has worked with other sequences, just not this new batch)
Ran this first and seemed to work on a HPC cluster:
Compressed file ended before the end-of-stream marker was reached
Debug info has been saved to /tmp/qiime2-q2cli-err-kik3isb7.log"
Navigating to /tmp (unsure if the /tmp is the same on the cluster as where the /tmp for the log file is) did not help either and file did not seem to exist (at least how I was searching), so I was wondering where the file would be located.
Good news! The log file's location is listed in the error message:
Your user account probably doesn't have permission to read other log files in that directory (/tmp), but if you ran cat /tmp/qiime2-q2cli-err-kik3isb7.log, you should be able to see the contents of that file. If you can't (permission error), I would then talk to your system administrator, that is a cluster config option, not anything we can do about it in QIIME 2. Also, that file might be gone now, so you will have to re-run, which will produce a new, differently named log. Lastly, if you don't want to deal with the log file, just run your command with the --verbose flag --- all the same log info will be logged to your terminal.
I can give you a hint here, check out the error message above:
That means you have a corrupt file somewhere in there! Can you run qiime tools validate demux-paired-end.qza? That will do a detailed check of the file. If that shows a similar error, then likely what happened is a file didn't transfer completely to your cluster, I would try to re-copy the files over from the sequencing center again.
I would like to run QIIME2 and it gives an error message. It might be because of the HPC allocation. Answering this question will help me understand how to run jobs on a shared cluster better and address time allocations, etc. I am going over more material for running jobs on a HPC shared computing resource (many researchers), and it mentions submitting jobs differently depending on job memory allocation:
MPI tasks with distributed memory
vs.
OpenMP with shared memory.
Which of the above two memory options would be used for converting qza to qzv files such as the below step for demux?
I moved this over to this thread — in the future please try not to post the same question (or nearly the same question) in multiple locations, it is hard for us to keep track of! Thanks for your understanding.
The QIIME 2 framework is neither of those memory models — it is single-host, shared memory (but not openmp enabled). There is no need for that with the framework, since all the framework does is handle communicating what a user asked for via an interface (like q2cli), and dispatching the command to the appropriate plugin.
QIIME 2 plugins could be either of those memory models you listed above, since plugins can wrap whatever tools they want. No tools come to mind though that take advantage of this.
Either way, I don’t think the memory model will have any bearing on the “corrupt file” issue you reported above. Did you run the qiime tools validate command I asked about above? Any findings there?
Can you run 