ITS extraction error - Missing sequence for record beginning on line 17

Pardal_Oblackovy · January 6, 2025, 10:38am

Pardal_Oblackovy:

qiime q2-itsxpress trim-pair
  --i-per-sample-sequences PL-02H_YT_demux_sequences.qza \
  --p-region ITS2  \
  --p-taxa F \
  --p-threads 80  \
  --o-trimmed trimmed-seqs.qza

Hello Colin @colinbrislawn , no, thank you for your for your perseverance!

So correct command shoul be:

qiime itsxpress trim-pair-output-unmerged \
  --i-per-sample-sequences PL-01H_demultiplexed.qza \
  --p-region ITS2 \
  --p-taxa F \
  --p-threads 60 \
  --o-trimmed PL-01H_trimmed-seqs.qza

right? I hope absolute paths are not mandatory when i run in from present dir...

Otuput of this command is currently:

Plugin error from itsxpress:

Command '['vsearch', '--fastq_mergepairs', '/tmp/qiime2/svecka/data/8f0ecfbc-41cb-443a-8f1d-885104ea0da5/data/NK2_AACGTGAT_L001_R1_001.fastq.gz', '--reverse', '/tmp/qiime2/svecka/data/8f0ecfbc-41cb-443a-8f1d-885104ea0da5/data/NK2_AACGTGAT_L001_R2_001.fastq.gz', '--fastqout', '/tmp/itsxpress_pwb4jx68/seq.fq', '--fastq_maxdiffs', '40', '--fastq_maxee', '2', '--threads', '60', '--fastq_allowmergestagger', '--fastq_qmax', '93']' returned non-zero exit status 1.

Debug info has been saved to /tmp/qiime2-q2cli-err-c2f942ot.log (pls see the att.)
qiime2-q2cli-err-c2f942ot.txt (3.5 KB)

regarding the disc size and writing permission:

(qiime2-amplicon-2024.2) svecka@svecserver:~/FRESH/FRESH_ITS/PL-01H/morning_glory_test$ df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv 915G 362G 515G 42% /

(qiime2-amplicon-2024.2) svecka@svecserver:~/FRESH/FRESH_ITS/PL-01H/morning_glory_test$ ls -ld .
drwxrwxr-x 6 svecka svecka 4096 Dec 18 15:05 .

Thank you very much and happy New Year

Karel

colinbrislawn · January 6, 2025, 3:11pm

Yes, that new command looks good with correct plugin name and backslashes on all lines except the last. I find that absolute paths are not usually needed, but can be helpful in some cases.

There error:
I opened up that file and saw this:
"Fatal error: Unable to open file for writing (/tmp/itsxpress_pwb4jx68/seq.fq)"

This is the next error we need to solve.

Filesystem                        Size Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv 915G 362G 515G   42% /

There is plenty of space left...

(qiime2-amplicon-2024.2) svecka@svecserver:~/FRESH/FRESH_ITS/PL-01H/morning_glory_test$ ls -ld .
drwxrwxr-x 6 svecka svecka 4096 Dec 18 15:05 .

So, looks like your group has Write permission for that folder.

Do you have write permissions in /tmp/? That's what the vsearch error showed so we should check there next.

Let me know what you find and if you have more questions!

Pardal_Oblackovy · January 6, 2025, 3:44pm

yes i should have... i created within the /tmp a new directory and testing script with:

#!/bin/bash
echo "Script executed successfully" in it and ran it succesfully.

-rw-rw-r-- 1 svecka svecka 15 Jan 6 15:29 testfile
-rwxrwxr-x 1 svecka svecka 49 Jan 6 15:34 testscript.sh

Both worked weird...

colinbrislawn · January 6, 2025, 6:05pm

Weird indeed!

Okay, maybe it's something else entirely!

How many reads are in each of your samples? Maybe one sample ends up with zero reads in it, which I have seen cause problems.

Pardal_Oblackovy · January 7, 2025, 9:48am

Reads counts should be OK

per-sample-fastq-counts (11).tsv (1.7 KB)

colinbrislawn · January 7, 2025, 3:59pm

Yeah, those look good!

I am out of ideas!

I'll leave this thread queued and let more knowledgeable team members take it from here.

Pardal_Oblackovy · January 8, 2025, 9:23am

I've run out of ideas also Now I'm afraid that my whole problem lies in some trivial mistake.

The data are demultiplexed sequences from Illumina Nextseq (2*250), the structure of the reads (length, counts, level of contamination by technical sequences, etc.) seems fine to me. When I manually ran shorter sections of reads in NCBI BLAST I was able to identify specific fungal organisms. Also my demultiplexed data can be processed in DADA2 directly (denoise-single works *"well" therfore i hypothetise my input FASTQs or .qza artefact, if you want, are OK

I will try to repeat the workflow on some test MOCK dataset.

Thank you for your help anyway @colinbrislawn @salias @gregcaporaso <3

I say "well" and not well because analysis can be performed but the proportion of filter passing sequences is ridiculously small -> thats also the reason why im trying ITS extraction priror the DADA analysis

colinvwood · January 8, 2025, 5:45pm

Hello @Pardal_Oblackovy,

It's unlikely to be the dataset that's the issue and likely to be related to the permissions on the /tmp directory. Have you tried setting the temporary directory to a different directory? You can accomplish this by setting an environment variable (which one can depend on your operating system, but likely either $TMP or $TMPDIR) prior to running the command.

Pardal_Oblackovy · January 9, 2025, 1:43pm

Hello @colinvwood Thank you for your advice, i ran it like (see below ... hope its correct). Redirecting tmp has worked but vsearch is still facing the same problems, i think:

(qiime2-amplicon-2024.2) svecka@svecserver:~/FRESH/FRESH_ITS/PL-01H/morning_glory_test$ export TMPDIR=~/FRESH/FRESH_ITS
(qiime2-amplicon-2024.2) svecka@svecserver:~/FRESH/FRESH_ITS/PL-01H/morning_glory_test$ qiime itsxpress trim-pair-output-unmerged \

--i-per-sample-sequences PL-01H_demultiplexed.qza
--p-region ITS2
--p-taxa F
--p-threads 60
--o-trimmed PL-01H_trimmed-seqs.qza
Plugin error from itsxpress:

Command '['vsearch', '--fastq_mergepairs', '/home/svecka/FRESH/FRESH_ITS/qiime2/svecka/data/8f0ecfbc-41cb-443a-8f1d-885104ea0da5/data/NK2_AACGTGAT_L001_R1_001.fastq.gz', '--reverse', '/home/svecka/FRESH/FRESH_ITS/qiime2/svecka/data/8f0ecfbc-41cb-443a-8f1d-885104ea0da5/data/NK2_AACGTGAT_L001_R2_001.fastq.gz', '--fastqout', '/home/svecka/FRESH/FRESH_ITS/itsxpress_t68zbpqn/seq.fq', '--fastq_maxdiffs', '40', '--fastq_maxee', '2', '--threads', '60', '--fastq_allowmergestagger', '--fastq_qmax', '93']' returned non-zero exit status 1.

Debug info has been saved to /home/svecka/FRESH/FRESH_ITS/qiime2-q2cli-err-o1jlp2iw.log
qiime2-q2cli-err-o1jlp2iw.txt (3.6 KB)

colinvwood · January 9, 2025, 5:11pm

Hello @Pardal_Oblackovy,

Can you run

stat $TMPDIR
whoami

and attach the output?

Pardal_Oblackovy · January 13, 2025, 9:29am

(qiime2-amplicon-2024.2) svecka@svecserver:~/FRESH/FRESH_ITS$ stat $TMPDIR
mi  File: /home/svecka/FRESH/FRESH_ITS
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: fd00h/64768d    Inode: 2385806     Links: 15
Access: (0775/drwxrwxr-x)  Uid: ( 1002/  svecka)   Gid: ( 1002/  svecka)
Access: 2025-01-09 13:11:31.197174290 +0000
Modify: 2025-01-09 13:05:48.748238847 +0000
Change: 2025-01-09 13:05:48.748238847 +0000
 Birth: 2024-10-19 12:31:54.034262031 +0000

(qiime2-amplicon-2024.2) svecka@svecserver:~/FRESH/FRESH_ITS$ whoami
svecka

colinvwood · January 13, 2025, 5:30pm

Hello @Pardal_Oblackovy,

Can you check each of the parent directories (/home, /home/svecka, and /home/svecka/FRESH/) for execute access? Your user needs execute permissions for each of these to write a file in the /home/svecka/FRESH/FRESH_ITS directory. If your user is the owner of each of these directories then you're looking for an x to be present in the first triplet of permission bits after the d so e.g. drwx... means you have execute permissions and e.g. drw-... means you don't have execute access.

colinvwood · January 13, 2025, 5:35pm

@Pardal_Oblackovy,

Another user had the same issue as you and was able to resolve it by reinstalling certain software, which might mean this is issue is actually a bug and not related to your directory permissions. You can check out that post here.

Pardal_Oblackovy · January 14, 2025, 12:05pm

Hello @colinvwood

(qiime2-amplicon-2024.2) svecka@svecserver:~/FRESH/FRESH_ITS$ test -x /home && echo "You have execute access to /home" || echo "No execute access to /home"
You have execute access to /home
e/sveck(qiime2-amplicon-2024.2) svecka@svecserver:~/FRESH/FRESH_ITS$ test -x /home/svecka && echo "You have execute access to /home/svecka" || echo "No execute access to /home/svecka"
You have execute access to /home/svecka
(qiime2-amplicon-2024.2) svecka@svecserver:~/FRESH/FRESH_ITS$ test -x /home/svecka/FRESH && echo "You have execute access to /home/svecka/FRESH" || echo "No execute access to /home/svecka/FRESH"
You have execute access to /home/svecka/FRESH
(qiime2-amplicon-2024.2) svecka@svecserver:~/FRESH/FRESH_ITS$

I think my access and rights should be guaranted...

Thank you for showing me the Need help: plugin error from ITSxpress - #6 by Lavendel ... i will also try to downgrade my ITSxpress 2.1.3 to ITSxpress 2.1.1 and ill let you know if it helped.

Adam_Rivers · January 14, 2025, 2:16pm

This was a bug introduced in v2.1.2. See Github Issue 60 for more details. It is now fixed in version 2.1.4.

Pardal_Oblackovy · January 15, 2025, 4:19pm

ITSexpress 2.1.1. gave me same result

Plugin error from itsxpress:

/tmp/q2-CasavaOneEightSingleLanePerSampleDirFmt-rjolube4/X06T_ATGCCTAA_L001_R2_001.fastq.gz is not a(n) FastqGzFormat file:

Missing sequence for record beginning on line 17

Debug info has been saved to /tmp/qiime2-q2cli-err-bs8swblf.log

I am trying ITS express 2.1.4. (THX @Adam_Rivers ) now, and.... this also returns me same error Weird...

But kind a small progress here:
From all my data i tested ive got this same Err log. Nevertheles today I tried repeat the extraction step on tutoruial demux data from Fungal ITS analysis tutorial and this worked. Total confusion

ir-output-unmerg(qiime2-amplicon-2023.9) svecka@svecserver:~/FRESH/FRESH_ITS/qiime2/tutotial$ qiime itsxpress trim-pair-output-unmerged
sample-s> --i-per-sample-sequences demux-trimmed.qza
on ITS> --p-region ITS1 \

--p-taxa F
--p-threads 60
--o-trimmed tutorial_ITS_extracted.qza
Saved SampleData[PairedEndSequencesWithQuality] to: tutorial_ITS_extracted.qza

soyeon_kim · January 16, 2025, 12:28pm

Hello, I have encountered the same error starting about a week ago.

I received this error message while performing ITS analysis in QIIME2 after importing ITS sequences and filtering for QC > 30 without removing primers using qiime cutadapt (Also Cutadapt standalone ver 4.9).
However, when I skipped QC filtering in qiime cutadapt (Also Cutadapt standalone ver 4.9) everything worked fine.

Env: QIIME-2024.2, CentOS 7.9, ITSxpress 2.1.1.dev3+g0ddb522.d20240917, vsearch v2.22.1_linux_x86_64, 1007.2GB RAM, 104 cores).

Although my QIIME and ITSxpress versions are different from yours, I wanted to share my experience in case it might be helpful.

Best regards,
Soyeon Kim

Pardal_Oblackovy · January 16, 2025, 12:40pm

Hello @soyeon_kim , thank you for your comment <3

I don't think it will be a solution for me. I didn't edit my sequences with Cutadapt...

But since I managed to extract ITS sequences from test data (Qiime2 ITS tutorial mock datasets) I think that the problem is not in the environment/programs themselves, but really in the data itself.

Thank you for your input in anyway

update 250115 16:45 CET

But i made a small experiment and tryed to treat my sequences by Cutadapt priror the ITS extraction to see what will happen...

svecka@svecserver:~/FRESH/FRESH_ITS/PL-01H/morning_glory_test$ qiime cutadapt trim-paired \

--i-demultiplexed-sequences PL-01H_demultiplexed.qza
--p-adapter-f ACAAGCTAACAAGCTA
--p-front-f GATGAAGAACGYAGYRAA
--p-adapter-r CTBTTVCCKCTTCACTCG
--p-front-r ACAAGCTA
--o-trimmed-sequences PL-01H_demux_trimmed.qza

(qiime2-amplicon-2023.9) svecka@svecserver:~/FRESH/FRESH_ITS/PL-01H/morning_glory_test$ qiime itsxpress trim-pair-output-unmerged \

--i-per-sample-sequences PL-01H_demux_trimmed.qza
--p-region ITS2
--p-taxa F
--p-threads 60
--o-trimmed PL-01H_ITS_extracted-seqs.qza
Plugin error from itsxpress:

/tmp/q2-CasavaOneEightSingleLanePerSampleDirFmt-otq2cci1/X01T_ACATTGGC_L001_R1_001.fastq.gz is not a(n) FastqGzFormat file:

Missing sequence for record beginning on line 5

Debug info has been saved to /tmp/qiime2-q2cli-err-9v7b0xxf.log
qiime2-q2cli-err-9v7b0xxf.txt (45.1 KB)