"Invalid value for 'input-path' Path does not exist" error on cluster

[mkc206 (Makaylee Crone)]

Hi,

I'm using the qiime2 module on the cluster system my university uses (ComputeCan) and am running into an error when I try to import my data. The error is "Invalid value for 'input-path' Path 'MetaData/manifest4.tsv' does not exist, but I'm using tab to finish the path when typing out the code, so I know it exists. I've also tried the longer path of /home/mkc206/scratch/MetaData/manifest4.tsv, but this doesn't work either. I'm concerned this has something to do with my environment set up.

Full code:

qiime tools import
--type 'SampleData[PairedEndSequencesWithQuality]'
--input-path MetaData/manifest4.tsv
--output-path 1_reads.qza
--input-format PairedEndFastqManifestPhred33V2

[cherman2 (Chloe Herman)]

Hi @mkc206,
That is weird!
Can you send in a screenshot of what you get when you ls in home/mkc206/scratch/MetaData/ ?

[mkc206 (Makaylee Crone)]

Thanks for responding!

So that's the "full" path, but because I'm submitting my code within scratch (which includes directory MetaData) I've been using --input-path MetaData/manifest4.tsv.

[cherman2 (Chloe Herman)]

Hi @mkc206,
Still weird, lets keep digging
I believe that the reason ls home/mkc206/scratch/MetaData/ didn't work is that it should be
ls /home/mkc206/scratch/MetaData/

What happens when you nano the .tsv file? I just want to make sure its not a permissions thing?

Are you currently in the QIIME 2 env?

[mkc206 (Makaylee Crone)]

Sorry forgot to add, yes I am in the qiime environment.

[mkc206 (Makaylee Crone)]

By nano do you mean open the file? I can use 'vim manifest4.tsv' to open and edit. I also ran the suggested path and this is the output:

[cherman2 (Chloe Herman)]

Hi @mkc206

@mkc206

By nano do you mean open the file?

Can you vim the file while in your qiime2 env? Nano is just another text editor!

[mkc206 (Makaylee Crone)]

yes, here is a screenshot of the first few rows:

[cherman2 (Chloe Herman)]

@mkc206
This is weird! I have never heard of a QIIME 2 env doing this. I think the next test is using test data and see if we can repeat the error?

Can you follow the steps for importing in the parkinsons mouse tutorial? Parkinson’s Mouse Tutorial — QIIME 2 2019.7.0 documentation
Let me know if you are able to successfully find the pd-mouse data!

[mkc206 (Makaylee Crone)]

Ok, I tried the first step of the Mouse tutorial by running this in directory mkc206 after loading the qiime2/2023.5 module:

apptainer exec qiime2.sif #the necessary code to make the module work on our server
qiime metadata tabulate \ #the code provided in the tutorial
--m-input-file metadata.tsv \ #this file is in /home/mkc206/projects/def-sanrehan/mkc206
--o-visualization metadata.qzv

and got the error:

Traceback (most recent call last):
File "/opt/conda/envs/qiime2-2023.7/bin/qiime", line 11, in
sys.exit(qiime())
File "/opt/conda/envs/qiime2-2023.7/lib/python3.8/site-packages/click/core.py", line 1157, in call
File "/opt/conda/envs/qiime2-2023.7/lib/python3.8/site-packages/click/core.py", line 1078, in main
File "/opt/conda/envs/qiime2-2023.7/lib/python3.8/site-packages/click/core.py", line 1682, in invoke
File "/opt/conda/envs/qiime2-2023.7/lib/python3.8/site-packages/click/core.py", line 1729, in resolve_command
File "/opt/conda/envs/qiime2-2023.7/lib/python3.8/site-packages/q2cli/commands.py", line 118, in get_command
File "/opt/conda/envs/qiime2-2023.7/lib/python3.8/site-packages/q2cli/commands.py", line 141, in init
File "/opt/conda/envs/qiime2-2023.7/lib/python3.8/site-packages/q2cli/util.py", line 231, in example_data_option
File "/opt/conda/envs/qiime2-2023.7/lib/python3.8/site-packages/q2cli/click/type.py", line 11, in
from qiime2.core.type.util import is_collection_type
File "/opt/conda/envs/qiime2-2023.7/lib/python3.8/site-packages/qiime2/init.py", line 9, in
from qiime2.sdk import Artifact, Visualization, ResultCollection
File "/opt/conda/envs/qiime2-2023.7/lib/python3.8/site-packages/qiime2/sdk/init.py", line 9, in
from .context import Context
File "/opt/conda/envs/qiime2-2023.7/lib/python3.8/site-packages/qiime2/sdk/context.py", line 13, in
File "/opt/conda/envs/qiime2-2023.7/lib/python3.8/site-packages/qiime2/sdk/parallel_config.py", line 15, in
File "/opt/conda/envs/qiime2-2023.7/lib/python3.8/site-packages/parsl/init.py", line 22, in
from parsl.app.app import bash_app, join_app, python_app
File "/opt/conda/envs/qiime2-2023.7/lib/python3.8/site-packages/parsl/app/app.py", line 12, in
from parsl.dataflow.dflow import DataFlowKernel
File "/opt/conda/envs/qiime2-2023.7/lib/python3.8/site-packages/parsl/dataflow/dflow.py", line 25, in
File "/opt/conda/envs/qiime2-2023.7/lib/python3.8/site-packages/parsl/channels/init.py", line 2, in
from parsl.channels.ssh.ssh import SSHChannel
File "/opt/conda/envs/qiime2-2023.7/lib/python3.8/site-packages/parsl/channels/ssh/ssh.py", line 5, in
File "/opt/conda/envs/qiime2-2023.7/lib/python3.8/site-packages/paramiko/init.py", line 22, in
from paramiko.transport import (
File "/opt/conda/envs/qiime2-2023.7/lib/python3.8/site-packages/paramiko/transport.py", line 93, in
File "/opt/conda/envs/qiime2-2023.7/lib/python3.8/site-packages/paramiko/dsskey.py", line 37, in
File "/opt/conda/envs/qiime2-2023.7/lib/python3.8/site-packages/paramiko/pkey.py", line 32, in
File "/opt/conda/envs/qiime2-2023.7/lib/python3.8/site-packages/bcrypt/init.py", line 34, in
from . import _bcrypt # noqa: I100
ImportError: /opt/conda/envs/qiime2-2023.7/lib/python3.8/site-packages/_cffi_backend.cpython-38-x86_64-linux-gnu.so: failed to map segment from shared object

[mkc206 (Makaylee Crone)]

Okay I may have figured it out - I thought I removed conda because I just learned that we are not supposed to use that on the server because it messes up the environment, but I see here in the error that it's still in my path! Do you know how I might get rid of it? I found instructions online saying to use ' export $PATH=/path/with/conda' but that didn't get rid of it. I also deleted the miniconda directory.

[colinbrislawn (Colin J Brislawn)]

Hi Makaylee,

I'm jumping in here because I also use qiime2 on HPC using modules (instead of the typical conda envs)

@mkc206

I found instructions online saying to use ' export $PATH=/path/with/conda' but that didn't get rid of it.

That would add conda to the path...

@mkc206

I also deleted the miniconda directory.

That a good start! My HPC team provides a qiime2 module and also a module for anaconda itself, so there may be multiple conda copies on the server.

This is the perfect time to talk to your HPC support team about how to do this.
They would know exactly how to avoid conflicts with conda and call the right version of Qiime2.
(Also, HPC want you to use their big computers so they tend to be nice )

[mkc206 (Makaylee Crone)]

Thank you! I did reach out to them and the problem wasn't with conda or qiime, but with apptainer. The -B bind paths need to be as follows below. I still don't quite understand why the file path needs to end in temp since the files are within the scratch directory, but this did solve the problem!

apptainer run -B /scratch/mkc206:/temp qimme2.sif qiime tools import
--type 'SampleData[PairedEndSequencesWithQuality]'
--input-path /temp/manifest3.tsv
--output-path /temp/1_reads.qza
--input-format PairedEndFastqManifestPhred33V2

Also thank you again to @cherman2!