Permission denied when importing FASTQ files

Hello,

I am attempting to import Casava 1.8 paired-end demultiplexed fastq files however I keep getting this output:

(base) [email protected]:~$ qiime tools import \

–type ‘SampleData[PairedEndSequencesWithQuality]’
–input-path /home/taf1g17
–input-format CasavaOneEightSingleLanePerSampleDirFmt
–output-path demux-paired-end.qza
Traceback (most recent call last):
File “/home/taf1g17/miniconda3/lib/python3.6/site-packages/q2cli/builtin/tools.py”, line 158, in import_data
view_type=input_format)
File “/home/taf1g17/miniconda3/lib/python3.6/site-packages/qiime2/sdk/result.py”, line 233, in import_data
md5sums = util.md5sum_directory(path)
File “/home/taf1g17/miniconda3/lib/python3.6/site-packages/qiime2/core/util.py”, line 126, in md5sum_directory
sums[os.path.relpath(path, start=directory)] = md5sum(path)
File “/home/taf1g17/miniconda3/lib/python3.6/site-packages/qiime2/core/util.py”, line 110, in md5sum
with open(str(filepath), mode=‘rb’) as fh:
PermissionError: [Errno 13] Permission denied: ‘/home/taf1g17/desktop.ini’

An unexpected error has occurred:

[Errno 13] Permission denied: ‘/home/taf1g17/desktop.ini’

See above for debug info.
(base) [email protected]:~$

I am using Ubuntu 20.04 LTS on Windows 10. I have tried running the Ubuntu terminal with administrator privileges but still get this error - is this something to do with the location of the files or just that I cannot access them? Please help!

Thanks!

Hi @taf1g17,

It appears that you tried to set the entire Home folder as the input path.

With CasavaOneEightSingleLanePerSampleDirFmt you’ll want to make a new folder / directory that only contains the sequence files.

If you want an easy way to share and use files between your Windows 10 and Ubuntu environment see these great posts:

-Mike

1 Like

Hello,

Thanks a lot for getting back to me! I have set up a new folder with ONLY sequence files in the newly created shared directory (as described in the first link provided), and set the input-path to this new location, however I am still getting the same error message:

Could there be another issue here or is this just a simple error I am making? Thanks.

Tom

Tried making some adjustments but still get this error:


I have no idea what I am doing wrong anymore.

Hi @taf1g17,

I noticed that you are within the share folder you set up based on the instructions I provided. Note this was only meant to have a place for you to easily access all of your files. However, you should make another folder, perhaps within the share folder that is labelled seqs-to-import or something. Copy only the *.fastq.qz files into the folder. It also appears that you have extra files other in that folder, i.e. .Zone.Identifier. I am not sure what these are…

Also, To keep things simple do not run the command from within the directory in which your sequences are located. I’d do the following:

For example, lets say we are in you home folder within WSL:
:computer:

taf1g17$: cd share
taf1g17:~/share$
taf1g17:~/share$ mkdir seqs-to-import
taf1g17:~/share$ cp *.fastq.gz seqs-to-import/
taf1g17:~/share$ qiime tools import \
                          >  --type 'SampleData[PariedEndSequencesWithQuality]'  \
                          >  --input-path seqs-to-import \
                          >  --input-format CasavaOneEightSingleLanePerSampleDirFmt \
                          >  --output-path imported-seqs.qza

Note: you do not need to provide the full file path, which is incorrect anyway as it is using backward-slashes. For more information see here. Also note that I simply provide a relative path.

Keep in mind the Windows Command Prompt does not work the same way as WSL Terminal. I’d recommend reading a few tutorials, like this one, about unix / linux systems. :man_teacher:

I also noticed that you seemed to have installed QIIME 2 into your base environment. I would highly advise against this as you may create potential issues. I’d follow the instructions as outlined for the WSL install and use the naming convention we describe there.

-I hope this helps! :slight_smile:
-Mike

Hello,

Thanks for your help!! I have created a separate folder (please) within which is the seq-to-import directory containing by fastq.gz files and a manifest.txt file. I think the data type was incorrect originally, however now I am getting a different error message (see screenshots) and cannot seem to import from that directory. What have I done wrong this time? Also, at this stage would you recommend uninstalling qiime2 from my base environment and then re-installing into a qiime2-specific working environment (as described in the tutorials?). Thanks
image

Tom

Hi @taf1g17,

Sorry, that was my fault. I should have noticed that the CasavaOneEightSingleLanePerSampleDirFmt is not the correct format, as the file names do not match the schema as outlined here. I think the easiest approach is to import using the Manifest Format.

-Mike

1 Like

Hello,

I have tried using the manifest format to import, but now I am getting this recursive error (see screenshots above)? Is this something to do with my directories/files or filepaths?

Thanks!!

Can you share the manifest file? I suspect the absolute file paths are incorrect.

Yeah sure, the file is below:

manifest.txt (1.6 KB)

Thanks

Hi @taf1g17,

This is very helpful thank you!

As I mentioned earlier is this thread, you can not use Windows-style file paths, but the file paths as WSL understands them. The best way to make sure you are using the correct Linux file paths is to cd into the folder where your sequence files are located, and type pwd. This will provide you with the absolute directory path. Do this within the WSL Terminal of course. :wink:

taf1g17$: cd share/seqs-to-import
taf1g17:~/share/seqs-to-import$ pwd  
    /home/taf1g17/share/seqs-to-import

Append this directory path to each of your files in your manifest file like so:

sample-id     forward-absolute-filepath     reverse-absolute-filepath
s1            /home/taf1g17/share/seqs-to-import/seqs_r1.fastq.gz     /home/taf1g17/share/seqs-to-import/seqs_r2.fastq.gz
...

Check out this UNIX / LINUX Tutorial.

-Hope this helps!
-Mike

1 Like

Hello,

Thanks again!

I have now appended the file names (see new manifest) however still get the same error:

manifest.txt (979 Bytes)

Is there anything else you can see?
Would it be recommended to try a complete uninstall of everything and start from scratch, or is this something that should be an easy fix? Sorry, I know I am probably doing something incredibly simple wrong but I cannot identify the issue!

Thanks,

Tom

No worries @taf1g17. This is because you need to set --input-path to point your manifest file. See the instructions I linked to earlier for importing Manifest formats. :man_teacher:

Hello,
Oh my god… It’s finally working (almost)!!! Although now I am encountering a new error, although I think that has more to do with the sequences rather than the code?? Could you take a look at the output I am getting and see what you think??
(base) [email protected]:~/please$ qiime tools import \

–type ‘SampleData[PairedEndSequencesWithQuality]’
–input-path /home/taf1g17/please/manifest.txt
–output-path paired-end-demux.qza
–input-format PairedEndFastqManifestPhred64V2
/home/taf1g17/miniconda3/lib/python3.6/site-packages/q2_types/per_sample_sequences/_transformer.py:243: UserWarning: Importing of PHRED 64 data is slow as it is converted internally to PHRED 33. Working with the imported data will not be slower than working with PHRED 33 data.
warnings.warn(_phred64_warning)
/home/taf1g17/miniconda3/lib/python3.6/site-packages/skbio/io/registry.py:557: ArgumentOverrideWarning: Best guess was: variant=‘illumina1.8’, continuing with user supplied: ‘illumina1.3’
ArgumentOverrideWarning)
Traceback (most recent call last):
File “/home/taf1g17/miniconda3/lib/python3.6/site-packages/q2cli/builtin/tools.py”, line 158, in import_data
view_type=input_format)
File “/home/taf1g17/miniconda3/lib/python3.6/site-packages/qiime2/sdk/result.py”, line 241, in import_data
validate_level=‘max’)
File “/home/taf1g17/miniconda3/lib/python3.6/site-packages/qiime2/sdk/result.py”, line 267, in _from_view
result = transformation(view, validate_level)
File “/home/taf1g17/miniconda3/lib/python3.6/site-packages/qiime2/core/transform.py”, line 70, in transformation
new_view = transformer(view)
File “/home/taf1g17/miniconda3/lib/python3.6/site-packages/q2_types/per_sample_sequences/_transformer.py”, line 246, in _26
single_end=False)
File “/home/taf1g17/miniconda3/lib/python3.6/site-packages/q2_types/per_sample_sequences/_util.py”, line 259, in _fastq_manifest_helper
fastq_copy_fn(input_fastq_fp, str(output_fastq_fp))
File “/home/taf1g17/miniconda3/lib/python3.6/site-packages/q2_types/per_sample_sequences/_util.py”, line 284, in _write_phred64_to_phred33
variant=‘illumina1.3’):
File “/home/taf1g17/miniconda3/lib/python3.6/site-packages/skbio/io/registry.py”, line 1161, in read
**kwargs)
File “/home/taf1g17/miniconda3/lib/python3.6/site-packages/skbio/io/registry.py”, line 506, in read
return (x for x in itertools.chain([next(gen)], gen))
File “/home/taf1g17/miniconda3/lib/python3.6/site-packages/skbio/io/registry.py”, line 531, in _read_gen
yield from reader(file, **kwargs)
File “/home/taf1g17/miniconda3/lib/python3.6/site-packages/skbio/io/registry.py”, line 1008, in wrapped_reader
yield from reader_function(fhs[-1], **kwargs)
File “/home/taf1g17/miniconda3/lib/python3.6/site-packages/skbio/io/format/fastq.py”, line 354, in _fastq_to_generator
qual_header)
File “/home/taf1g17/miniconda3/lib/python3.6/site-packages/skbio/io/format/fastq.py”, line 523, in _parse_quality_scores
phred_offset=phred_offset))
File “/home/taf1g17/miniconda3/lib/python3.6/site-packages/skbio/io/format/_base.py”, line 35, in _decode_qual_to_phred
% (phred_range[0], phred_range[1]))
ValueError: Decoded Phred score is out of range [0, 62].

An unexpected error has occurred:

Decoded Phred score is out of range [0, 62].

See above for debug info.
(base) [email protected]:~/please$

Again, thank you so much - this is vital for my dissertation and I have nowhere else to turn!

Tom

Awesome @taf1g17, we are getting closer! I think we just need to try a different --input-format. Try using --input-format PairedEndFastqManifestPhred33V2. If you would like to see what the available options are you can use the following commands:

qiime tools import --show-importable-types
qiime tools import --show-importable-formats

Here is a wiki on the various FASTQ formats.

Thank you so much - now up and running and working through the different work flows! Honestly, you’ve saved me.

Take it easy,

Tom

1 Like