q2-qemistree [Errno 2] No such file or directory: 'tmp/qiime2-archive/data/csi-output'

Community Plugin Support
,
1

Hello, everyone,

I'm running qiime2-2019.7 in a conda environment, and I'm trying to run Qemistree using qiime2 by it's q2-qemistree plugin. (seems like not many people are using q2-qemistree).

I'm following the q2-qemistree tutorial step by step but I got stuck at generating a hierarchy of molecules (earlier steps are working well and I checked all inputs and outputs).

When trying to run 'qiime qemistree make-hierarchy' like below,

qiime qemistree make-hierarchy \
  --i-csi-results fingerprints.qza \
  --i-feature-tables feature-table.qza \
  --o-tree qemistree.qza \
  --o-feature-table feature-table-hashed.qza \
  --o-feature-data feature-data.qza

I got the error message like below.

Traceback (most recent call last)
File "/home/user/anaconda2/envs/qiime2-2019.7/lib/python3.6/site-packages/q2cli/commands.py", line 327, in call
results = action(**arguments)
File "</home/user/anaconda2/envs/qiime2-2019.7/lib/python3.6/site-packages/decorator.py:decorator-gen-217>", line 2, in make_hierarchy
File "/home/user/anaconda2/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/sdk/action.py", line 240, in bound_callable
output_types, provenance)
File "/home/user/anaconda2/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/sdk/action.py", line 383, in callable_executor
output_views = self._callable(**view_args)
File "/home/user/anaconda2/envs/qiime2-2019.7/lib/python3.6/site-packages/q2_qemistree/_hierarchy.py", line 126, in make_hierarchy
qc_properties, metric)
File "/home/user/anaconda2/envs/qiime2-2019.7/lib/python3.6/site-packages/q2_qemistree/_process_fingerprint.py", line 95, in process_csi_results
collated_fps = collate_fingerprint(csi_result, qc_properties, metric)
File "/home/user/anaconda2/envs/qiime2-2019.7/lib/python3.6/site-packages/q2_qemistree/_process_fingerprint.py", line 28, in collate_fingerprint
fpfoldrs = os.listdir(csi_result)
FileNotFoundError: [Errno 2] No such file or directory: '/data/tmp/qiime2-archive-397d8bha/072dce7a-0c53-46ed-b2a4-a01013fe1828/data/csi-output'

I have tried to overwrite the tmp directory by
export TMPDIR ='/data'
(I also checked that the variable is set by running 'echo $TMPDIR')

However, it still gives the same type of error
([Errno 2] No such file or directory: '/data/qiime2-archive...')

I browsed TMPDIR directory and keep looking at it while q2-qemistree is running,
and I saw some temporary directories were created but they were gone in few seconds.
Then I checked the directory permissions for TMPDIR but couldn't fined any issues.

Any help would be greatly appreciated.

3

Hi @tinmad and welcome to the forum! :wave:

Can you do a quick sanity check for me and just confirm that the .qza files you are passing into the qemistree make-hierarchy are in fact present in your working directory? Prior to trying to run the command again, if you could copy/paste the following commands and their outputs, that would be great:

  • pwd
  • ls -l

Please share the output of the above commands, then try running make-hierarchy again. Another thing you can try is to pass in the absolute paths to those .qza files. If you are on linux you can use readlink -f fingerprints.qza to print the absolute path of a file.

1 Like
4

Thanks @andrewsanchez!

@tinmad, for what its worth, I am unable to reproduce the tutorial results on my development machine, as well. There is some atypical stuff going on with this plugin related to how the sirius tool is included, as well as the additional configuration of the tmp dir (for java) - perhaps @Anupriya_Tripathi can weigh in?

There appears to be a problem in one of the earlier steps, qiime qemistree predict-fingerprints generates a fingerprints file with no data in it, just stdout and stderr logs:

Screen Shot 2021-01-26 at 7.20.33 AM

This lines up with the error message you shared above, there really is no csi-output subdirectory present in the output data.

The contents of the stderr log:

Picked up _JAVA_OPTIONS:  -Djava.io.tmpdir=/path-to-some-dir/ -Xms16G -Xmx64G
INFO  07:17:25 - Sirius Workspace Successfull initialized at: /Users/matthew/.sirius
INFO  07:17:25 - You run SIRIUS 4.0.1 (build 10)
INFO  07:17:25 - Sirius was compiled with the following ILP solvers: GLPK-v1.7.0 (included), Gurobi-v7.5.2, CPLEX-v12.8
INFO  07:17:25 - Treebuilder priorities are: [GUROBI, CPLEX, GLPK]
JNA Warning: IOException removing temporary files: JNA temporary directory '/path-to-some-dir' does not exist
Exception in thread "Thread-0" java.lang.NoClassDefFoundError: Could not initialize class de.unijena.bioinf.sirius.core.ApplicationCore
	at de.unijena.bioinf.ms.cli.SiriusCLIApplication.lambda$main$0(SiriusCLIApplication.java:17)
	at java.base/java.lang.Thread.run(Thread.java:834)

It looks like the error is related to the tmpdir param I provided in an earlier step, but really the predict-fingerprints command should have just failed on us, rather that producing invalid results.

Check out this note from the tutorial:

Note : /path-to-some-dir/ should be a directory where you have write permissions and sufficient storage space. We use -Xms16G and -Xmx64G as the minimum and maximum heap size for Java virtual machine (JVM). If left blank, q2-qemistree will use default JVM flags.

Hopefully @Anupriya_Tripathi can comment on this for us. Thanks!

1 Like
5

For what its worth, by updating --p-java-flags "-Djava.io.tmpdir=/path-to-some-dir/ -Xms16G -Xmx64G" to point to the same tmp dir as my QIIME 2 environment worked as expected.

2 Likes
12

Hi, @tinmad! Thanks for the question. As @thermokarst suggests, could you please try updating the JVM flags and re-running the tutorial from qiime qemistree predict-fingerprint command? As Qemistree predicts fingerprints by querying the CSI:FingerID server, could you also make sure you have internet access while you’re running this step?

Before running qiime qemistree make-hierarchy, please confirm that the fingerprints have been predicted successfully (else make-hierarchy will fail). You can do this by exporting fingerprint.qza. If successful, you should see a folder named csi-output in your exported artifact.

Please let us know if this works for you. Thanks!

3 Likes
13

Excuse me ,I meet the same problem:

I use both the example file and my own file ,but encounter the same error,saying
"plugin error from qemistree:

[Errno 2] No such file or directory: ‘/tmp/qiime2-archive-3pqdbn9p/70cc36ba-ddc2-4c09-9003-e163d59dfb26/data/csi-output’

Debug info has been saved to /tmp/qiime2-q2cli-err-ti_39i0d.log"

Similarily, I also change the TMPDIR to the /tmp BUt did’nt solve

14

Hi all, I have had this problem since last week. I tried running Qemistree using Qiime2 in a conda environment on my laptop, as well as remotely on a supercomputer to no avail. qiime qemistree make-hierarchy would not run in either environment because qiime qemistree predict-fingerprint did not make a csi-output, only a file with stdout and stderr logs but no data, like @thermokarst pointed out.

This may because there is an issue the Sirius server, or in the communication between Qemistree and Sirius. Can you comment on this @Anupriya_Tripathi ?? Thanks!

15

Hey Colin, I experienced the same issue, is it in any way solved for you? Thanks!

16

Hi Ahmet, I still have the same problem. I think there’s a problem with Sirius connecting to qemistree. So, I’m using the Sirius GUI interface to run CSI:fingerprints so I can get the annotated list of compounds from CANOPUS for now.

1 Like
17

Thanks Colin, I realized that my rerank-molecular-formulas also doesn't give any output. So basically the solution to these is running Sirius/CSI/zodiac locally and importing them somehow to qiime formats to use within qemistree. I don't know though if there's a way in qemistree to run everything locally without using Sirius servers but I'll ask the developers as well if that's possible.