Erno 28, Erno 1 when importing Cassava 1.8

[ana_RODILES (Ana)]

Hi qiimers,

After to run the whole Moving picture tutorial with my own Cassava 1.8 data set (19 samples) with @thermokarst help (Thanks again!), I have been trying with a bigger data set (90 samples). I am using qiime2-2017.11 on a Microsoft azure virtual machine. I am absolutely stuck with these error messages, please help! Many thanks in advance.

The first error I got when running qiime tools import:
"(qiime2) mothur@Microbiome:/mnt$ qiime tools import --type 'SampleData[PairedEndSequencesWithQuality]' --input-path /mnt/fastqfiles/GATC-04/I02304-01/fastq --source-format CasavaOneEightSingleLanePerSampleDirFmt --output-path /mnt/fastqfiles/GATC-04/I02304-01/fastq/demux-paired-end.qza
Traceback (most recent call last):
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/q2cli/tools.py", line 116, in import_data view_type=source_format)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/qiime2/sdk/result.py", line 211, in import_data return cls.from_view(type, view, view_type, provenance_capture)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/qiime2/sdk/result.py", line 236, in _from_view result = transformation(view)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/qiime2/core/transform.py", line 59, in transformation new_view = transformer(view)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/q2_types/per_sample_sequences/_transformer.py", line 76, in _4 dirfmt, SingleLanePerSamplePairedEndFastqDirFmt)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/q2_types/per_sample_sequences/_transformer.py", line 52, in _single_lane_per_sample_fastq_helper read_number=read_number)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/qiime2/plugin/model/directory_format.py", line 87, in write_data result.path._move_or_copy(self.path_maker(**kwargs))
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/qiime2/core/path.py", line 35, in _move_or_copy return shutil.copy(str(self), str(other))
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/shutil.py", line 241, in copy copyfile(src, dst, follow_symlinks=follow_symlinks)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/shutil.py", line 122, in copyfile copyfileobj(fsrc, fdst)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/shutil.py", line 82, in copyfileobj fdst.write(buf)
OSError: [Errno 28] No space left on device
An unexpected error has occurred:
[Errno 28] No space left on device
See above for debug info."

Then I created a /tmp folder:
sudo mkdir /home/mothur/output/temp/
export TMPDIR=/home/mothur/output/temp/
echo $TMPDIR

An run again with this new error:
"(qiime2) mothur@Microbiome:/mnt$ qiime tools import --type 'SampleData[PairedEndSequencesWithQuality]' --input-path /mnt/fastqfiles/GATC-04/bz2files/abm/gz --source-format CasavaOneEightSingleLanePerSampleDirFmt --output-path /mnt/output/I02304-01/demux-paired-end.qza
Traceback (most recent call last):
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/q2cli/tools.py", line 116, in import_data view_type=source_format)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/qiime2/sdk/result.py", line 211, in import_data return cls.from_view(type, view, view_type, provenance_capture)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/qiime2/sdk/result.py", line 236, in _from_view result = transformation(view)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/qiime2/core/transform.py", line 59, in transformation new_view = transformer(view)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/q2_types/per_sample_sequences/_transformer.py", line 76, in _4 dirfmt, SingleLanePerSamplePairedEndFastqDirFmt)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/q2_types/per_sample_sequences/_transformer.py", line 52, in _single_lane_per_sample_fastq_helper read_number=read_number)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/qiime2/plugin/model/directory_format.py", line 87, in write_data result.path._move_or_copy(self.path_maker(**kwargs))
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/qiime2/core/path.py", line 35, in _move_or_copy return shutil.copy(str(self), str(other))
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/shutil.py", line 242, in copy copymode(src, dst, follow_symlinks=follow_symlinks)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/shutil.py", line 144, in copymode chmod_func(dst, stat.S_IMODE(st.st_mode))
PermissionError: [Errno 1] Operation not permitted: '/home/mothur/output/temp/q2-SingleLanePerSamplePairedEndFastqDirFmt-f7237m49/L4A01_1_L001_R1_001.fastq.gz'
An unexpected error has occurred:
[Errno 1] Operation not permitted: '/home/mothur/output/temp/q2-SingleLanePerSamplePairedEndFastqDirFmt-f7237m49/L4A01_1_L001_R1_001.fastq.gz'
See above for debug info."

So I also try with sudo after apply or not "sudo chmod 777 $TMPDIR":

"(qiime2) mothur@Microbiome:/mnt$ sudo qiime tools import --type 'SampleData[PairedEndSequencesWithQuality]' --input-path /mnt/fastqfiles/GATC-04/bz2files/abm/gz --source-format CasavaOneEightSingleLanePerSampleDirFmt --output-path /mnt/output/I02304-01/demux-paired-end.qza
/usr/bin/qiime: 52: exec: /usr/lib/qiime/bin/tools.py: not found"

Thanks a lot.

[thermokarst]

Hi @ana_RODILES!

Reading through your session info, it looks like you ran out of disk space (maybe just tmp space?), and then you created a new tmp dir and pointed QIIME 2 at it, but it failed. This definitely looks like a permission issue to me - why did you create your new tmp dir with sudo? How about you create a new tmp dir without using elevated permission, and try again?

In general, I wouldn't recommend using sudo unless you know you need it - you can wind up with all sorts of issues (for example, in the very last error you reported above, the use of sudo is actually changing your environment, dropping your QIIME 2 env, and trying to use a QIIME 1 deployment - this is probably not what you want to have happen). Let us know if that gets you moving in the right direction!

[ana_RODILES (Ana)]

Hola @thermokarst!

Many thanks for your reply. I started again and created a folder 'temp' in /output, then in /fastqfiles, then out in /tmp and run again the commands, but I got again and again the same error but the last one, please look below:

"(qiime2) mothur@Microbiome:/mnt/output$ mkdir temp
(qiime2) mothur@Microbiome:/mnt/output$ export TMPDIR=/mnt/output/temp/
(qiime2) mothur@Microbiome:/mnt/output$ echo TMPDIR /mnt/output/temp/ (qiime2) mothur@Microbiome:/mnt/output qiime tools import --type 'SampleData[PairedEndSequencesWithQuality]' --input-path /mnt/fastqfiles/GATC-04/bz2files/abm/gz --source-format CasavaOneEightSingleLanePerSampleDirFmt --output-path /mnt/output/I02304-01/demux-paired-end.qza
Traceback (most recent call last):
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/q2cli/tools.py", line 116, in import_data
view_type=source_format)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/qiime2/sdk/result.py", line 211, in import_data
return cls.from_view(type, view, view_type, provenance_capture)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/qiime2/sdk/result.py", line 236, in _from_view
result = transformation(view)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/qiime2/core/transform.py", line 59, in transformation
new_view = transformer(view)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/q2_types/per_sample_sequences/_transformer.py", line 76, in _4
dirfmt, SingleLanePerSamplePairedEndFastqDirFmt)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/q2_types/per_sample_sequences/_transformer.py", line 52, in _single_lane_per_sample_fastq_helper
read_number=read_number)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/qiime2/plugin/model/directory_format.py", line 87, in write_data
result.path._move_or_copy(self.path_maker(**kwargs))
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/qiime2/core/path.py", line 35, in _move_or_copy
return shutil.copy(str(self), str(other))
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/shutil.py", line 242, in copy
copymode(src, dst, follow_symlinks=follow_symlinks)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/shutil.py", line 144, in copymode
chmod_func(dst, stat.S_IMODE(st.st_mode))
PermissionError: [Errno 1] Operation not permitted: '/mnt/output/temp/q2-SingleLanePerSamplePairedEndFastqDirFmt-le0ii0rc/L4A01_1_L001_R1_001.fastq.gz'

An unexpected error has occurred:
[Errno 1] Operation not permitted: '/mnt/output/temp/q2-SingleLanePerSamplePairedEndFastqDirFmt-le0ii0rc/L4A01_1_L001_R1_001.fastq.gz'
See above for debug info."

I also tried to change --output path with no luck:

"(qiime2) mothur@Microbiome:/mnt$ echo TMPDIR /mnt/fastqfiles/temp/ (qiime2) mothur@Microbiome:/mnt qiime tools import --type 'SampleData[PairedEndSequencesWithQuality]' --input-path /mnt/fastqfiles/GATC-04/bz2files/abm/gz --source-format CasavaOneEightSingleLanePerSampleDirFmt --output-path /mnt/fastqfiles/GATC-04/bz2files/abm/gz/demux-paired-end.qza
Traceback (most recent call last):
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/q2cli/tools.py", line 116, in import_data
view_type=source_format)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/qiime2/sdk/result.py", line 211, in import_data
return cls.from_view(type, view, view_type, provenance_capture)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/qiime2/sdk/result.py", line 236, in _from_view
result = transformation(view)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/qiime2/core/transform.py", line 59, in transformation
new_view = transformer(view)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/q2_types/per_sample_sequences/_transformer.py", line 76, in _4
dirfmt, SingleLanePerSamplePairedEndFastqDirFmt)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/q2_types/per_sample_sequences/_transformer.py", line 52, in _single_lane_per_sample_fastq_helper
read_number=read_number)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/qiime2/plugin/model/directory_format.py", line 87, in write_data
result.path._move_or_copy(self.path_maker(**kwargs))
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/qiime2/core/path.py", line 35, in _move_or_copy
return shutil.copy(str(self), str(other))
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/shutil.py", line 242, in copy
copymode(src, dst, follow_symlinks=follow_symlinks)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/shutil.py", line 144, in copymode
chmod_func(dst, stat.S_IMODE(st.st_mode))
PermissionError: [Errno 1] Operation not permitted: '/mnt/fastqfiles/temp/q2-SingleLanePerSamplePairedEndFastqDirFmt-ol4z3al3/L4A01_1_L001_R1_001.fastq.gz'
An unexpected error has occurred:
[Errno 1] Operation not permitted: '/mnt/fastqfiles/temp/q2-SingleLanePerSamplePairedEndFastqDirFmt-ol4z3al3/L4A01_1_L001_R1_001.fastq.gz'
See above for debug info."

I also try to use this /tmp out of mnt/ and got a different error! With it was the first error I got when tried to run the import tools without defining a temp folder:

"(qiime2) mothur@Microbiome:/mnt$ export TMPDIR=/tmp
(qiime2) mothur@Microbiome:/mnt$ echo TMPDIR /tmp (qiime2) mothur@Microbiome:/mnt qiime tools import --type 'SampleData[PairedEndSequencesWithQuality]' --input-path /mnt/fastqfiles/GATC-04/bz2files/abm/gz --source-format CasavaOneEightSingleLanePerSampleDirFmt --output-path /mnt/fastqfiles/GATC-04/bz2files/abm/gz/demux-paired-end.qza
Traceback (most recent call last):
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/q2cli/tools.py", line 116, in import_data
view_type=source_format)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/qiime2/sdk/result.py", line 211, in import_data
return cls.from_view(type, view, view_type, provenance_capture)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/qiime2/sdk/result.py", line 236, in _from_view
result = transformation(view)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/qiime2/core/transform.py", line 59, in transformation
new_view = transformer(view)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/q2_types/per_sample_sequences/_transformer.py", line 76, in _4
dirfmt, SingleLanePerSamplePairedEndFastqDirFmt)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/q2_types/per_sample_sequences/_transformer.py", line 52, in _single_lane_per_sample_fastq_helper
read_number=read_number)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/qiime2/plugin/model/directory_format.py", line 87, in write_data
result.path._move_or_copy(self.path_maker(**kwargs))
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/site-packages/qiime2/core/path.py", line 35, in _move_or_copy
return shutil.copy(str(self), str(other))
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/shutil.py", line 241, in copy
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/shutil.py", line 122, in copyfile
copyfileobj(fsrc, fdst)
File "/home/mothur/miniconda3/envs/qiime2/lib/python3.5/shutil.py", line 82, in copyfileobj
fdst.write(buf)
OSError: [Errno 28] No space left on device
An unexpected error has occurred:
[Errno 28] No space left on device
See above for debug info."

I know I have space in there (below) but doesn't let me use it...

Filesystem 1K-blocks Used Available Use% Mounted on
udev 28831056 0 28831056 0% /dev
tmpfs 5770232 8820 5761412 1% /run
/dev/sda1 29711408 19476304 10218720 66% /
tmpfs 28851140 0 28851140 0% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
tmpfs 28851140 0 28851140 0% /sys/fs/cgroup
none 64 0 64 0% /etc/network/interfaces.dynamic.d
/dev/sdb1 115465696 61056 109516288 1% /mnt
tmpfs 5770232 0 5770232 0% /run/user/1000
/dev/sdc1 131979748 57554512 67698020 46% /home/mothur/runenv
/dev/sdd1 528315064 37022728 464432460 8% /home/mothur/runenv2
//mothurdata2.file.core.windows.net/fastqfiles 5368709120 297890368 5070818752 6% /mnt/fastqfiles
//mothurdata2.file.core.windows.net/output 5368709120 13361600 5355347520 1% /mnt/output

Many thanks in advance!
Ana.

[ana_RODILES (Ana)]

Hi @thermokarst,

I tried again after checking ls -l, placing the analysis in a place without sudo permission (no 'root') and it works finally!! Great news for a Friday!!

Many thanks for your help,

Ana.