qiime2-import data from non-working directory

User Support
1

Hello, qiime2 users community!

I have the following set-up:

  • a huge collection of .fastq files which I would like to process with the dada2 pipeline
  • a remote cluster of servers with the separated storage where those files are stored, and the working machines for computing.

Question: Is it possible to import the data from the directory which is not currently used (i.e. from the storage to the working directory)?

It seems that it was not possible before, according to this post from jwdebelius.

The command I use in the $cwd:

qiime demux summarize \
--i-data $storage_dir/test.qza \
--o-visualization test.qzv

The message error:

(1/2) Invalid value for '--i-data':
  'q2-import/$storage_dir/test.qza' is not a valid filepath
(2/2) Invalid value for '--o-visualization': Directory
  'q2-import/$storage_dir' does not exist, cannot save
  'test.qzv' into it.

If I run the same command having the data (test.qza) in the $cwd, everything works just fine.

Thank you!

3

Hi @Oleg, welcome to the :qiime2: forum!

This is what I would recommend as your course of action - work through each grouping of files that you have by changing your $CWD and handling them one location at a time. You could move all of your files to a single location as a secondary option, but I assume you've separated them onto different partitions/storage locations due to the large number of files you have (as you mention above).

Hopefully this helps!

Cheers,
Liz

5

Hi, Liz!

Thanks for your response, let me clarify my question because I think your reply touches another thing.

I have a .qza file in the directory: home/storage/paired-end-demux.qza
and I have a iPython notebook in the directory: home/computation/q2.ipynb

Is it possible to access paired-end-demux.qza with qiime2 script written in q2.ipynb?
i.e. run the following being in home/computation/q2.ipynb

qiime demux summarize \
--i-data q2-import/home/storage/paired-end-demux.qza \
--o-visualization test.qzv

Hope I could explain my issue better now, thank you!

7

Hi @Oleg,

Thanks for providing that clarification! Just as a heads up, I didn't have time to get to this before the long weekend, but will follow up with you on Tuesday after the holiday. Thanks for your patience!

Cheers,
Liz

1 Like
8

2 off-topic replies have been split into a new topic: I have a problem when I execute the qiime tools import

Please keep replies on-topic in the future.

9

Hi @Oleg,

Thanks for your patience here! Happy to address your follow up below:

There is nothing within the QIIME 2 Framework that would prevent this - but if you are running into a permissions issue while having your file and script in different locations (i.e. errors along the line of: not a valid filepath, file not found, not allowed, etc), this is most likely something you'll need to reach out to your sysadmin about (if you are not the admin for the cluster of servers you are working with).

Best of luck here, hopefully this helps point you in the right direction!

Cheers,
Liz

1 Like
11

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.