Error while importing files from tutorial

importing
import

#1

I was following “Importing data” tutorial, however I encoutered a problem while trying to import data (actually the first task in tutorial).

I tried to import provided data (emp-single-end-sequences) with this command:

(qiime2-2018.8) [email protected]:~/qiime2-importing-tutorial$ qiime tools import \
type EMPSingleEndSequences \
--input-path emp-single-end-sequences \
--output-path emp-single-end-sequences.qza

However, this results in:

Error: Invalid value for "--input-path": Path "emp-single-end-sequences" does not exist.

I am in right working directory (according to tutorial) and I have downloaded all necessary data. I am running qiime2 natively (from conda enviroment) in Ubuntu 16.04.1 LTS, file system ext4.

Other files cause the same error to happen.

On a separate physical machine, when I created a virtual machine and installed Ubuntu 18.04, everything worked fine.

What could be the cause of this error and how can I solve this?

Edit: I have created emp-single-end-sequences.
Edit2: Some question mistakes corrected.


Modifying QIIME2 execution wrapper in Conda
Modifying QIIME2 execution wrapper in Conda
(Nicholas Bokulich) #2

Hi @h7405244,

This means you missed an earlier step in the tutorial (making the directory), or you are working in a new directory that you weren’t working in prior. Try again from the start, paying attention to each command.

You are missing this step:
mkdir emp-single-end-sequences

Make sure you are saving the sequence files to that directory.

I hope that helps!


#3

Hello @Nicholas_Bokulich

I have created the directory and entered it, I just haven’t mentioned it in my question, sorry.


(Nicholas Bokulich) #4

You want to create that directory and save the sequence files into that directory but you should not be in that directory when you run that command.

You will know you are in the correct directory if the following command produces information, not an error:

ls emp-single-end-sequences

#5

Sorry, I read it incorrectly and didn\t understand.

Step-by-step:
I created qiime2-importing-tutorial directory.
I made qiime2-importing-tutorial my working directory.
I created emp-single-end-sequences directory.
I downloaded barcodes.fastq.gz and sequences.fastq.gz into emp-single-end-sequences directory.
I ran:

(qiime2-2018.8) [email protected]:~/qiime2-importing-tutorial$ qiime tools import \
type EMPSingleEndSequences \
--input-path emp-single-end-sequences \
--output-path emp-single-end-sequences.qza

and I am getting the aforementioned error.


(Nicholas Bokulich) #6

Thanks for clarifying @h7405244! And sorry if my questions seemed arbitrarily dumb — always best to work through the basics before tackling deeper issues.

So could you please let us know:

  1. What platform you are running QIIME 2 on
  2. The output of ls emp-single-end-sequences

#7

No no, questions are allright, it was my mistake, I did not read them correctly.

1. question

$ cat /etc/issue
Ubuntu 16.04.1 LTS \n \l

$conda info
     active environment : qiime2-2018.8
    active env location : /home/user/miniconda3/envs/qiime2-2018.8
            shell level : 1
       user config file : /home/user/.condarc
 populated config files : /home/user/.condarc
          conda version : 4.5.11
    conda-build version : not installed
         python version : 3.7.0.final.0
       base environment : /home/user/miniconda3  (writable)
           channel URLs : https://conda.anaconda.org/bonsai-team/linux-64
                          https://conda.anaconda.org/bonsai-team/noarch
                          https://conda.anaconda.org/agbiome/linux-64
                          https://conda.anaconda.org/agbiome/noarch
                          https://conda.anaconda.org/bioconda/linux-64
                          https://conda.anaconda.org/bioconda/noarch
                          https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/linux-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/pro/linux-64
                          https://repo.anaconda.com/pkgs/pro/noarch
                          https://conda.anaconda.org/r/linux-64
                          https://conda.anaconda.org/r/noarch
          package cache : /home/user/miniconda3/pkgs
                          /home/user/.conda/pkgs
       envs directories : /home/user/miniconda3/envs
                          /home/user/.conda/envs
               platform : linux-64
             user-agent : conda/4.5.11 requests/2.19.1 CPython/3.7.0 Linux/4.4.0-137-generic ubuntu/16.04 glibc/2.23
                UID:GID : 1003:1003
             netrc file : None
           offline mode : False

$qiime info
System versions
Python version: 3.5.5
QIIME 2 release: 2018.8
QIIME 2 version: 2018.8.0
q2cli version: 2018.8.0

Installed plugins
alignment: 2018.8.0
composition: 2018.8.0
cutadapt: 2018.8.0
dada2: 2018.8.0
deblur: 2018.8.0
demux: 2018.8.0
diversity: 2018.8.0
emperor: 2018.8.0
feature-classifier: 2018.8.0
feature-table: 2018.8.0
gneiss: 2018.8.0
longitudinal: 2018.8.0
metadata: 2018.8.0
phylogeny: 2018.8.0
quality-control: 2018.8.0
quality-filter: 2018.8.0
sample-classifier: 2018.8.0
taxa: 2018.8.0
types: 2018.8.0
vsearch: 2018.8.0

Application config directory
/home/qiime2/q2cli

Getting help
To get help with QIIME 2, visit https://qiime2.org

$mount
/dev/sda1 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)

$pwd
/home/user/qiime2-importing-tutorial

2. question

$ls emp-single-end-sequences
barcodes.fastq.gz  sequences.fastq.gz

(Nicholas Bokulich) #14

Hi @h7405244,
Your system info looks all in order and the files clearly exist in the correct location. This should work. The most likely explanation seems to be that you are in the wrong working directory when you attempt to run the import command. Could you please run all the following and copy/paste all of the inputs/outputs?

pwd
mkdir qiime2-importing-tutorial
cd qiime2-importing-tutorial
mkdir emp-single-end-sequences
wget \
  -O "emp-single-end-sequences/barcodes.fastq.gz" \
  "https://data.qiime2.org/2018.8/tutorials/moving-pictures/emp-single-end-sequences/barcodes.fastq.gz"
wget \
  -O "emp-single-end-sequences/sequences.fastq.gz" \
  "https://data.qiime2.org/2018.8/tutorials/moving-pictures/emp-single-end-sequences/sequences.fastq.gz"
ls emp-single-end-sequences
qiime tools import \
  --type EMPSingleEndSequences \
  --input-path emp-single-end-sequences \
  --output-path emp-single-end-sequences.qza

#15

Hello @Nicholas_Bokulich,

Error still occurs, all inputs and outputs:

(qiime2-2018.8) [email protected]:~$ pwd
/home/user
(qiime2-2018.8) [email protected]:~$ mkdir qiime2-importing-tutorial
(qiime2-2018.8) [email protected]:~$ cd qiime2-importing-tutorial
(qiime2-2018.8) [email protected]:~/qiime2-importing-tutorial$ mkdir emp-single-end-sequences
(qiime2-2018.8) [email protected]:~/qiime2-importing-tutorial$ wget \
>   -O "emp-single-end-sequences/barcodes.fastq.gz" \
>   "https://data.qiime2.org/2018.8/tutorials/moving-pictures/emp-single-end-sequences/barcodes.fastq.gz"
--2018-11-06 14:58:38--  https://data.qiime2.org/2018.8/tutorials/moving-pictures/emp-single-end-sequences/barcodes.fastq.gz
Resolving data.qiime2.org... 104.27.171.158, 104.27.170.158, 2606:4700:30::681b:aa9e, ...
Connecting to data.qiime2.org|104.27.171.158|:443... connected.
HTTP request sent, awaiting response... 302 FOUND
Location: https://s3-us-west-2.amazonaws.com/qiime2-data/2018.8/tutorials/moving-pictures/emp-single-end-sequences/barcodes.fastq.gz [following]
--2018-11-06 14:58:38--  https://s3-us-west-2.amazonaws.com/qiime2-data/2018.8/tutorials/moving-pictures/emp-single-end-sequences/barcodes.fastq.gz
Resolving s3-us-west-2.amazonaws.com... 52.218.212.8
Connecting to s3-us-west-2.amazonaws.com|52.218.212.8|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3783785 (3.6M) [application/x-gzip]
Saving to: ‘emp-single-end-sequences/barcodes.fastq.gz’

emp-single-end-sequences/barcodes.fastq.gz        100%[===========================================================================================================>]   3.61M  2.42MB/s    in 1.5s

2018-11-06 14:58:40 (2.42 MB/s) - ‘emp-single-end-sequences/barcodes.fastq.gz’ saved [3783785/3783785]

(qiime2-2018.8) [email protected]:~/qiime2-importing-tutorial$ wget \
>   -O "emp-single-end-sequences/sequences.fastq.gz" \
>   "https://data.qiime2.org/2018.8/tutorials/moving-pictures/emp-single-end-sequences/sequences.fastq.gz"
--2018-11-06 14:58:40--  https://data.qiime2.org/2018.8/tutorials/moving-pictures/emp-single-end-sequences/sequences.fastq.gz
Resolving data.qiime2.org... 104.27.170.158, 104.27.171.158, 2606:4700:30::681b:aa9e, ...
Connecting to data.qiime2.org|104.27.170.158|:443... connected.
HTTP request sent, awaiting response... 302 FOUND
Location: https://s3-us-west-2.amazonaws.com/qiime2-data/2018.8/tutorials/moving-pictures/emp-single-end-sequences/sequences.fastq.gz [following]
--2018-11-06 14:58:41--  https://s3-us-west-2.amazonaws.com/qiime2-data/2018.8/tutorials/moving-pictures/emp-single-end-sequences/sequences.fastq.gz
Resolving s3-us-west-2.amazonaws.com... 52.218.212.8
Connecting to s3-us-west-2.amazonaws.com|52.218.212.8|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 25303756 (24M) [binary/octet-stream]
Saving to: ‘emp-single-end-sequences/sequences.fastq.gz’

emp-single-end-sequences/sequences.fastq.gz       100%[===========================================================================================================>]  24.13M  1.42MB/s    in 22s

2018-11-06 14:59:03 (1.11 MB/s) - ‘emp-single-end-sequences/sequences.fastq.gz’ saved [25303756/25303756]

(qiime2-2018.8) [email protected]:~/qiime2-importing-tutorial$ ls emp-single-end-sequences
barcodes.fastq.gz  sequences.fastq.gz
(qiime2-2018.8) [email protected]:~/qiime2-importing-tutorial$ qiime tools import \
>   --type EMPSingleEndSequences \
>   --input-path emp-single-end-sequences \
>   --output-path emp-single-end-sequences.qza
Usage: qiime tools import [OPTIONS]

Error: Invalid value for "--input-path": Path "emp-single-end-sequences" does not exist.

(Matthew Ryan Dillon) #17

@h7405244 - can you please run the following command and paste the output here:

env

Please keep in mind sensitive information (tokens, paths, etc) might be present, so take a moment to review before posting.


#18

@thermokarst
Here is the output - I omitted some information that were clearly not connected to this problem:

TERM=xterm
SHELL=/bin/bash
PYTHONNOUSERSITE=/home/user/miniconda3/envs/qiime2-2018.8/lib/python*/site-packages/
CONDA_SHLVL=1
PERL5LIB=/usr/local/perl/lib/perl5
CONDA_PROMPT_MODIFIER=(qiime2-2018.8)
PERL_MB_OPT=--install_base /usr/local/perl
LC_ALL=en_US.UTF-8
USER=user
CONDA_EXE=/home/user/miniconda3/bin/conda
PATH=/home/user/miniconda3/envs/qiime2-2018.8/bin:/home/user/bin:/home/user/.local/bin:/home/user/miniconda3/bin:/home/user/miniconda3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/data/bin:/usr/local/bin:/usr/local/perl:/data/bin:/snap/bin
CONDA_PREFIX=/home/user/miniconda3/envs/qiime2-2018.8
PWD=/home/user
MPLBACKEND=Agg
SHLVL=1
HOME=/home/user
LANGUAGE=en_US:en
CONDA_PYTHON_EXE=/home/user/miniconda3/bin/python
LOGNAME=user
CONDA_DEFAULT_ENV=qiime2-2018.8
R_LIBS_USER=/home/user/miniconda3/envs/qiime2-2018.8/lib/R/library/
LESSCLOSE=/usr/bin/lesspipe %s %s
PERL_MM_OPT=INSTALL_BASE=/usr/local/perl
_=/usr/bin/env

(Nicholas Bokulich) #19

(Matthew Ryan Dillon) #20

Bummer, I was hoping we would see a PYTHONPATH env var, or something along those lines…

To be honest, I have no idea what is wrong — this is the first time in 2+ years on the QIIME 2 Forum that I have seen a case like this, I am really not sure what to do for you. Is it possible that you have emdashes or endashes in one or more of your commands? Does your shell allow for tab-completion? If so, what happens if you try and tab-complete the paths? Is this machine maintained by some kind of sysadmin? If so, maybe you should think about getting them involved. This doesn’t seem like it is a problem with QIIME 2 specifically, but rather, your QIIME 2 environment, or, your deployment of conda. Maybe try removing the env and trying again? Or, try reinstalling conda altogether. Sorry, none of those answers are super satisfying.

One other test, does this work?

wget https://docs.qiime2.org/2018.8/data/tutorials/moving-pictures/table.qza
qiime tools validate table.qza

You should see a message like this:

Artifact table.qza appears to be valid at level=max.

Also, in your qiime2-importing-tutorial dir, can you run ls -lah and provide the results?


(Matthew Ryan Dillon) #21

(system) #22

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.