q2-qemistree [Errno 2] No such file or directory: 'tmp/qiime2-archive/data/csi-output'

Hello, everyone,

I’m running qiime2-2019.7 in a conda environment, and I’m trying to run Qemistree using qiime2 by it’s q2-qemistree plugin. (seems like not many people are using q2-qemistree).

I’m following the q2-qemistree tutorial step by step but I got stuck at generating a hierarchy of molecules (earlier steps are working well and I checked all inputs and outputs).

When trying to run ‘qiime qemistree make-hierarchy’ like below,

qiime qemistree make-hierarchy \
  --i-csi-results fingerprints.qza \
  --i-feature-tables feature-table.qza \
  --o-tree qemistree.qza \
  --o-feature-table feature-table-hashed.qza \
  --o-feature-data feature-data.qza

I got the error message like below.

Traceback (most recent call last)
File “/home/user/anaconda2/envs/qiime2-2019.7/lib/python3.6/site-packages/q2cli/commands.py”, line 327, in call
results = action(**arguments)
File “</home/user/anaconda2/envs/qiime2-2019.7/lib/python3.6/site-packages/decorator.py:decorator-gen-217>”, line 2, in make_hierarchy
File “/home/user/anaconda2/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/sdk/action.py”, line 240, in bound_callable
output_types, provenance)
File “/home/user/anaconda2/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/sdk/action.py”, line 383, in callable_executor
output_views = self._callable(**view_args)
File “/home/user/anaconda2/envs/qiime2-2019.7/lib/python3.6/site-packages/q2_qemistree/_hierarchy.py”, line 126, in make_hierarchy
qc_properties, metric)
File “/home/user/anaconda2/envs/qiime2-2019.7/lib/python3.6/site-packages/q2_qemistree/_process_fingerprint.py”, line 95, in process_csi_results
collated_fps = collate_fingerprint(csi_result, qc_properties, metric)
File “/home/user/anaconda2/envs/qiime2-2019.7/lib/python3.6/site-packages/q2_qemistree/_process_fingerprint.py”, line 28, in collate_fingerprint
fpfoldrs = os.listdir(csi_result)
FileNotFoundError: [Errno 2] No such file or directory: ‘/data/tmp/qiime2-archive-397d8bha/072dce7a-0c53-46ed-b2a4-a01013fe1828/data/csi-output’

I have tried to overwrite the tmp directory by
export TMPDIR =’/data’
(I also checked that the variable is set by running ‘echo $TMPDIR’)

However, it still gives the same type of error
([Errno 2] No such file or directory: ‘/data/qiime2-archive…’)

I browsed TMPDIR directory and keep looking at it while q2-qemistree is running,
and I saw some temporary directories were created but they were gone in few seconds.
Then I checked the directory permissions for TMPDIR but couldn’t fined any issues.

Any help would be greatly appreciated.

Hi @tinmad and welcome to the forum! :wave:

Can you do a quick sanity check for me and just confirm that the .qza files you are passing into the qemistree make-hierarchy are in fact present in your working directory? Prior to trying to run the command again, if you could copy/paste the following commands and their outputs, that would be great:

  • pwd
  • ls -l

Please share the output of the above commands, then try running make-hierarchy again. Another thing you can try is to pass in the absolute paths to those .qza files. If you are on linux you can use readlink -f fingerprints.qza to print the absolute path of a file.

1 Like

Thanks @andrewsanchez!

@tinmad, for what its worth, I am unable to reproduce the tutorial results on my development machine, as well. There is some atypical stuff going on with this plugin related to how the sirius tool is included, as well as the additional configuration of the tmp dir (for java) - perhaps @Anupriya_Tripathi can weigh in?

There appears to be a problem in one of the earlier steps, qiime qemistree predict-fingerprints generates a fingerprints file with no data in it, just stdout and stderr logs:

Screen Shot 2021-01-26 at 7.20.33 AM

This lines up with the error message you shared above, there really is no csi-output subdirectory present in the output data.

The contents of the stderr log:

Picked up _JAVA_OPTIONS:  -Djava.io.tmpdir=/path-to-some-dir/ -Xms16G -Xmx64G
INFO  07:17:25 - Sirius Workspace Successfull initialized at: /Users/matthew/.sirius
INFO  07:17:25 - You run SIRIUS 4.0.1 (build 10)
INFO  07:17:25 - Sirius was compiled with the following ILP solvers: GLPK-v1.7.0 (included), Gurobi-v7.5.2, CPLEX-v12.8
INFO  07:17:25 - Treebuilder priorities are: [GUROBI, CPLEX, GLPK]
JNA Warning: IOException removing temporary files: JNA temporary directory '/path-to-some-dir' does not exist
Exception in thread "Thread-0" java.lang.NoClassDefFoundError: Could not initialize class de.unijena.bioinf.sirius.core.ApplicationCore
	at de.unijena.bioinf.ms.cli.SiriusCLIApplication.lambda$main$0(SiriusCLIApplication.java:17)
	at java.base/java.lang.Thread.run(Thread.java:834)

It looks like the error is related to the tmpdir param I provided in an earlier step, but really the predict-fingerprints command should have just failed on us, rather that producing invalid results.

Check out this note from the tutorial:

Note : /path-to-some-dir/ should be a directory where you have write permissions and sufficient storage space. We use -Xms16G and -Xmx64G as the minimum and maximum heap size for Java virtual machine (JVM). If left blank, q2-qemistree will use default JVM flags.

Hopefully @Anupriya_Tripathi can comment on this for us. Thanks!

1 Like

For what its worth, by updating --p-java-flags "-Djava.io.tmpdir=/path-to-some-dir/ -Xms16G -Xmx64G" to point to the same tmp dir as my QIIME 2 environment worked as expected.

2 Likes

Hi, @tinmad! Thanks for the question. As @thermokarst suggests, could you please try updating the JVM flags and re-running the tutorial from qiime qemistree predict-fingerprint command? As Qemistree predicts fingerprints by querying the CSI:FingerID server, could you also make sure you have internet access while you’re running this step?

Before running qiime qemistree make-hierarchy, please confirm that the fingerprints have been predicted successfully (else make-hierarchy will fail). You can do this by exporting fingerprint.qza. If successful, you should see a folder named csi-output in your exported artifact.

Please let us know if this works for you. Thanks!

2 Likes