Qiime2 TMPDIR Errno 45

[mist]

Hello,
I am working on my Mac with fastq data from an external hard drive and I want to start my analysis with qiime import. Since the size of the files is large, temp files take a lot of space. I changed the TMPDIR to a file in the hard drive with "export TMPDIR=/filepath" and got Errno 45 after performing qiime import. To clarify, I wanted to have the fastq files from the hard drive as input and store the output and temp files in another file in the same external hard drive. Have you ever been in this situation and what was your work around?

Thanks in advance!

[Oddant1 (Anthony Simard)]

Hello @mist, can you please post the exact series of commands you ran and the exact error you encountered? Thank you.

[mist]

Hello @Oddant1 ,

Here are the series of commands and the error.

export TMPDIR='/Volumes/hard_drive_name/TEMP' (TEMP is a directory I created for this job)
echo TMPDIR gives the exact path to TEMP file
cd /Volumes/hard_drive_name
sh qiime_import.sh

qiime_import.sh contains:
qiime tools import
--type 'SampleData[PairedEndSequencesWithQuality]'
--input-path /Volumes/hard_drive_name/manifest_file_name.tsv
--output-path /Volumes/hard_drive_name/demux_paired/paired_end_demux_file_name.qza
--input-format PairedEndFastqManifestPhred33V2

Traceback (most recent call last):
File "~/opt/anaconda3/envs/qiime2-metagenome-2024.5/lib/python3.9/site-packages/q2cli/builtin/tools.py", line 852, in _import
artifact = qiime2.sdk.Artifact.import_data(
File "~/opt/anaconda3/envs/qiime2-metagenome-2024.5/lib/python3.9/site-packages/qiime2/sdk/result.py", line 332, in import_data
return cls.from_view(type, view, view_type, provenance_capture,
File "~/opt/anaconda3/envs/qiime2-metagenome-2024.5/lib/python3.9/site-packages/qiime2/sdk/result.py", line 360, in _from_view
result = transformation(view, validate_level)
File "~/opt/anaconda3/envs/qiime2-metagenome-2024.5/lib/python3.9/site-packages/qiime2/core/transform.py", line 70, in transformation
new_view = transformer(view)
File "~/opt/anaconda3/envs/qiime2-metagenome-2024.5/lib/python3.9/site-packages/q2_types/per_sample_sequences/_transformer.py", line 243, in _16
return _fastq_manifest_helper_partial(old_fmt, _copy_with_compression,
File "~/opt/anaconda3/envs/qiime2-metagenome-2024.5/lib/python3.9/site-packages/q2_types/per_sample_sequences/_util.py", line 270, in _fastq_manifest_helper
fastq_copy_fn(input_fastq_fp, str(output_fastq_fp))
File "~/opt/anaconda3/envs/qiime2-metagenome-2024.5/lib/python3.9/site-packages/q2_types/per_sample_sequences/_util.py", line 238, in _copy_with_compression
qiime2.util.duplicate(src, dst)
File "~/opt/anaconda3/envs/qiime2-metagenome-2024.5/lib/python3.9/site-packages/qiime2/util.py", line 89, in duplicate
os.link(src, dst)
OSError: [Errno 45] Operation not supported: '/Volumes/hard_drive_name/sequence_name.fastq.gz' -> '/Volumes/hard_drive_name/TEMP/q2-SingleLanePerSamplePairedEndFastqDirFmt-4m126kz9/temp_sequence_name.fastq.gz'

An unexpected error has occurred:

[Errno 45] Operation not supported: '/Volumes/hard_drive_name/sequence_name.fastq.gz' -> '/Volumes/hard_drive_name/TEMP/q2-SingleLanePerSamplePairedEndFastqDirFmt-4m126kz9/temp_sequence_name.fastq.gz'

See above for debug info.

[Oddant1 (Anthony Simard)]

@mist, I believe the problem here is that we attempt to create a symlink when possible instead of fully copying the files because creating a symlink is more efficient. Unfortunately, it isn't generally possible to create symlinks on removable media like external harddrives and usb sticks.

I think it would be straightforward to account for this in the code, but I'm not sure if there is a workaround possible on your end.