Cutadapt thinks my fastq.gz files aren't fastq.gz

Erin_dfo · March 8, 2018, 1:38pm

I am very new to using Qiime2, and I don’t have any coding experience.

I have single end, already demultiplexed reads produced by an Ion Torrent PGM. I have been able to import these reads using the fastq manifest format, and everything went fine. I was also able to import them as Casava format. The below error happens with both formats.

I have tried to trim my primers using this command:

qiime cutadapt trim-single --i-demultiplexed-sequences MC1_run1_demux.qza --p-adapter CTTCCGGTACACTTACCATG --p-front ACACCGCCCGTCACTCT --p-error-rate 0.1 --o-trimmed-sequences MC1_run1_Teleo_trimmed.qza --verbose

and I get this error message:

Plugin error from cutadapt:

  /tmp/q2-CasavaOneEightSingleLanePerSampleDirFmt-40fz3afg/Teleo_1_L001_R1_001.fastq.gz is not a(n) FastqGzFormat file:

  Missing sequence for record beginning on line 5

See above for debug info

Here is the complete output:

Running external command line application. This may print messages to stdout and/or stderr.
The commands to be run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: cutadapt --cores 1 --error-rate 0.1 --times 1 --overlap 3 -o /tmp/q2-CasavaOneEightSingleLanePerSampleDirFmt-fxl7g674/MC1_run1_12S_1_L001_R1_001.fastq.gz --adapter CTTCCGGTACACTTACCATG --front ACACCGCCCGTCACTCT /tmp/qiime2-archive-25f11crf/0a91b822-d856-4f7b-9489-add0061b5fed/data/MC1_run1_12S_1_L001_R1_001.fastq.gz

This is cutadapt 1.15 with Python 3.5.4
Command line parameters: --cores 1 --error-rate 0.1 --times 1 --overlap 3 -o /tmp/q2-CasavaOneEightSingleLanePerSampleDirFmt-fxl7g674/MC1_run1_12S_1_L001_R1_001.fastq.gz --adapter CTTCCGGTACACTTACCATG --front ACACCGCCCGTCACTCT /tmp/qiime2-archive-25f11crf/0a91b822-d856-4f7b-9489-add0061b5fed/data/MC1_run1_12S_1_L001_R1_001.fastq.gz
Running on 1 core
Trimming 2 adapters with at most 10.0% errors in single-end mode ...
Finished in 5.26 s (36 us/read; 1.65 M reads/minute).

=== Summary ===

Total reads processed:                 145,009
Reads with adapters:                       246 (0.2%)
Reads written (passing filters):       145,009 (100.0%)

Total basepairs processed:    18,525,079 bp
Total written (filtered):     18,524,316 bp (100.0%)

=== Adapter 1 ===

Sequence: CTTCCGGTACACTTACCATG; Type: regular 3'; Length: 20; Trimmed: 235 times.

No. of allowed errors:
0-9 bp: 0; 10-19 bp: 1; 20 bp: 2

Bases preceding removed adapters:
  A: 19.6%
  C: 40.9%
  G: 30.6%
  T: 8.9%
  none/other: 0.0%

Overview of removed sequences
length	count	expect	max.err	error counts
3	220	2265.8	0	220
4	10	566.4	0	10
5	1	141.6	0	1
6	4	35.4	0	4

=== Adapter 2 ===

Sequence: ACACCGCCCGTCACTCT; Type: regular 5'; Length: 17; Trimmed: 11 times.

No. of allowed errors:
0-9 bp: 0; 10-17 bp: 1

Overview of removed sequences
length	count	expect	max.err	error counts
3	10	2265.8	0	10
4	1	566.4	0	1


Command: cutadapt --cores 1 --error-rate 0.1 --times 1 --overlap 3 -o /tmp/q2-CasavaOneEightSingleLanePerSampleDirFmt-fxl7g674/MC1_run1_MiFish_2_L001_R1_001.fastq.gz --adapter CTTCCGGTACACTTACCATG --front ACACCGCCCGTCACTCT /tmp/qiime2-archive-25f11crf/0a91b822-d856-4f7b-9489-add0061b5fed/data/MC1_run1_MiFish_2_L001_R1_001.fastq.gz

This is cutadapt 1.15 with Python 3.5.4
Command line parameters: --cores 1 --error-rate 0.1 --times 1 --overlap 3 -o /tmp/q2-CasavaOneEightSingleLanePerSampleDirFmt-fxl7g674/MC1_run1_MiFish_2_L001_R1_001.fastq.gz --adapter CTTCCGGTACACTTACCATG --front ACACCGCCCGTCACTCT /tmp/qiime2-archive-25f11crf/0a91b822-d856-4f7b-9489-add0061b5fed/data/MC1_run1_MiFish_2_L001_R1_001.fastq.gz
Running on 1 core
Trimming 2 adapters with at most 10.0% errors in single-end mode ...
Finished in 5.33 s (47 us/read; 1.27 M reads/minute).

=== Summary ===

Total reads processed:                 112,529
Reads with adapters:                       581 (0.5%)
Reads written (passing filters):       112,529 (100.0%)

Total basepairs processed:    21,585,797 bp
Total written (filtered):     21,583,924 bp (100.0%)

=== Adapter 1 ===

Sequence: CTTCCGGTACACTTACCATG; Type: regular 3'; Length: 20; Trimmed: 483 times.

No. of allowed errors:
0-9 bp: 0; 10-19 bp: 1; 20 bp: 2

Bases preceding removed adapters:
  A: 17.2%
  C: 12.2%
  G: 48.9%
  T: 21.7%
  none/other: 0.0%

Overview of removed sequences
length	count	expect	max.err	error counts
3	374	1758.3	0	374
4	98	439.6	0	98
5	7	109.9	0	7
6	1	27.5	0	1
7	2	6.9	0	2
10	1	0.1	1	0 1

=== Adapter 2 ===

Sequence: ACACCGCCCGTCACTCT; Type: regular 5'; Length: 17; Trimmed: 98 times.

No. of allowed errors:
0-9 bp: 0; 10-17 bp: 1

Overview of removed sequences
length	count	expect	max.err	error counts
3	98	1758.3	0	98


Command: cutadapt --cores 1 --error-rate 0.1 --times 1 --overlap 3 -o /tmp/q2-CasavaOneEightSingleLanePerSampleDirFmt-fxl7g674/MC1_run1_Teleo_0_L001_R1_001.fastq.gz --adapter CTTCCGGTACACTTACCATG --front ACACCGCCCGTCACTCT /tmp/qiime2-archive-25f11crf/0a91b822-d856-4f7b-9489-add0061b5fed/data/MC1_run1_Teleo_0_L001_R1_001.fastq.gz

This is cutadapt 1.15 with Python 3.5.4
Command line parameters: --cores 1 --error-rate 0.1 --times 1 --overlap 3 -o /tmp/q2-CasavaOneEightSingleLanePerSampleDirFmt-fxl7g674/MC1_run1_Teleo_0_L001_R1_001.fastq.gz --adapter CTTCCGGTACACTTACCATG --front ACACCGCCCGTCACTCT /tmp/qiime2-archive-25f11crf/0a91b822-d856-4f7b-9489-add0061b5fed/data/MC1_run1_Teleo_0_L001_R1_001.fastq.gz
Running on 1 core
Trimming 2 adapters with at most 10.0% errors in single-end mode ...
Finished in 1.56 s (29 us/read; 2.08 M reads/minute).

=== Summary ===

Total reads processed:                  54,129
Reads with adapters:                    53,627 (99.1%)
Reads written (passing filters):        54,129 (100.0%)

Total basepairs processed:     5,085,118 bp
Total written (filtered):      1,378,531 bp (27.1%)

=== Adapter 1 ===

Sequence: CTTCCGGTACACTTACCATG; Type: regular 3'; Length: 20; Trimmed: 36399 times.

No. of allowed errors:
0-9 bp: 0; 10-19 bp: 1; 20 bp: 2

Bases preceding removed adapters:
  A: 0.0%
  C: 0.2%
  G: 0.0%
  T: 0.1%
  none/other: 99.6%

Overview of removed sequences
length	count	expect	max.err	error counts
3	11	845.8	0	11
4	3	211.4	0	3
5	1	52.9	0	1
24	1	0.0	2	0 1
25	166	0.0	2	126 27 13
26	124	0.0	2	94 19 11
27	120	0.0	2	93 15 12
28	83	0.0	2	59 14 10
29	52	0.0	2	33 12 7
30	55	0.0	2	36 10 9
31	61	0.0	2	54 3 4
32	52	0.0	2	34 13 5
33	65	0.0	2	56 2 7
34	39	0.0	2	35 3 1
35	59	0.0	2	45 10 4
36	48	0.0	2	42 4 2
37	257	0.0	2	238 17 2
38	72	0.0	2	50 20 2
39	54	0.0	2	43 8 3
40	69	0.0	2	64 3 2
41	109	0.0	2	95 7 7
42	129	0.0	2	116 8 5
43	145	0.0	2	130 13 2
44	94	0.0	2	81 11 2
45	107	0.0	2	100 3 4
46	131	0.0	2	112 15 4
47	108	0.0	2	90 12 6
48	134	0.0	2	116 15 3
49	132	0.0	2	104 20 8
50	109	0.0	2	96 12 1
51	150	0.0	2	130 16 4
52	74	0.0	2	46 25 3
53	97	0.0	2	82 11 4
54	178	0.0	2	160 14 4
55	79	0.0	2	53 21 5
56	108	0.0	2	78 21 9
57	86	0.0	2	57 24 5
58	65	0.0	2	44 16 5
59	62	0.0	2	45 10 7
60	66	0.0	2	48 16 2
61	55	0.0	2	39 16
62	43	0.0	2	30 11 2
63	72	0.0	2	61 9 2
64	57	0.0	2	35 18 4
65	44	0.0	2	29 12 3
66	43	0.0	2	30 11 2
67	71	0.0	2	47 18 6
68	49	0.0	2	43 6
69	111	0.0	2	96 12 3
70	60	0.0	2	42 17 1
71	82	0.0	2	70 7 5
72	49	0.0	2	37 10 2
73	43	0.0	2	36 7
74	51	0.0	2	44 5 2
75	38	0.0	2	25 12 1
76	37	0.0	2	30 4 3
77	53	0.0	2	44 7 2
78	50	0.0	2	37 11 2
79	40	0.0	2	34 4 2
80	40	0.0	2	35 3 2
81	64	0.0	2	56 4 4
82	45	0.0	2	35 8 2
83	108	0.0	2	102 6
84	43	0.0	2	38 5
85	54	0.0	2	49 4 1
86	72	0.0	2	66 3 3
87	61	0.0	2	50 9 2
88	62	0.0	2	48 10 4
89	73	0.0	2	61 8 4
90	53	0.0	2	44 9
91	46	0.0	2	39 4 3
92	55	0.0	2	48 5 2
93	59	0.0	2	52 4 3
94	39	0.0	2	29 10
95	116	0.0	2	107 8 1
96	148	0.0	2	140 4 4
97	251	0.0	2	241 8 2
98	523	0.0	2	453 23 47
99	989	0.0	2	888 78 23
100	18487	0.0	2	18351 120 16
101	8385	0.0	2	8297 83 5
102	877	0.0	2	796 75 6
103	335	0.0	2	278 46 11
104	209	0.0	2	167 34 8
105	123	0.0	2	86 33 4
106	65	0.0	2	36 27 2
107	45	0.0	2	23 21 1
108	52	0.0	2	29 21 2
109	36	0.0	2	21 11 4
110	41	0.0	2	22 16 3
111	17	0.0	2	10 6 1
112	16	0.0	2	11 3 2
113	16	0.0	2	10 5 1
114	10	0.0	2	7 3
115	10	0.0	2	8 1 1
116	3	0.0	2	3
117	6	0.0	2	6
118	7	0.0	2	4 0 3
119	7	0.0	2	6 1
120	7	0.0	2	6 1
121	8	0.0	2	7 1
122	7	0.0	2	4 2 1
123	5	0.0	2	5
124	8	0.0	2	8
125	3	0.0	2	3
126	3	0.0	2	0 1 2
127	3	0.0	2	3
129	3	0.0	2	2 0 1
130	4	0.0	2	3 1
131	3	0.0	2	3
132	2	0.0	2	0 1 1
133	2	0.0	2	0 2
135	4	0.0	2	4
136	3	0.0	2	3
137	3	0.0	2	3
138	3	0.0	2	3
140	2	0.0	2	2
141	4	0.0	2	4
142	3	0.0	2	1 2
143	4	0.0	2	3 1
144	2	0.0	2	2
145	5	0.0	2	4 1
147	1	0.0	2	1
148	1	0.0	2	1
149	1	0.0	2	1
150	1	0.0	2	1
151	1	0.0	2	1
152	1	0.0	2	1
153	2	0.0	2	2
157	2	0.0	2	2
162	1	0.0	2	1
163	1	0.0	2	1
164	3	0.0	2	3
166	1	0.0	2	1
170	1	0.0	2	1
173	1	0.0	2	1
175	2	0.0	2	2
181	1	0.0	2	0 0 1
182	1	0.0	2	1
183	1	0.0	2	1
187	1	0.0	2	1
190	1	0.0	2	1
193	1	0.0	2	1
198	2	0.0	2	2
199	1	0.0	2	1
203	1	0.0	2	1
206	1	0.0	2	1
207	1	0.0	2	1
208	1	0.0	2	1
215	1	0.0	2	1
218	1	0.0	2	1
219	1	0.0	2	1
221	1	0.0	2	1
225	1	0.0	2	1
258	1	0.0	2	1
266	1	0.0	2	1
297	3	0.0	2	3
298	1	0.0	2	1
299	1	0.0	2	1
300	1	0.0	2	1
301	1	0.0	2	1
309	1	0.0	2	1
312	1	0.0	2	1
338	1	0.0	2	1
347	1	0.0	2	1
348	2	0.0	2	2
349	1	0.0	2	0 0 1
354	2	0.0	2	1 1
360	1	0.0	2	1

=== Adapter 2 ===

Sequence: ACACCGCCCGTCACTCT; Type: regular 5'; Length: 17; Trimmed: 17228 times.

No. of allowed errors:
0-9 bp: 0; 10-17 bp: 1

Overview of removed sequences
length	count	expect	max.err	error counts
9	1	0.2	0	1
10	1	0.1	1	1
11	5	0.0	1	5
12	9	0.0	1	9
13	79	0.0	1	75 4
14	193	0.0	1	183 10
15	90	0.0	1	62 28
16	485	0.0	1	378 107
17	15298	0.0	1	15169 129
18	1058	0.0	1	57 1001
19	5	0.0	1	2 3
24	1	0.0	1	1
30	1	0.0	1	1
33	1	0.0	1	1
42	1	0.0	1	1

Traceback (most recent call last):
  File "/home/qiime2/miniconda/envs/qiime2-2017.12/lib/python3.5/site-packages/qiime2/plugin/model/file_format.py", line 24, in validate
    self._validate_(level)
  File "/home/qiime2/miniconda/envs/qiime2-2017.12/lib/python3.5/site-packages/q2_types/per_sample_sequences/_format.py", line 159, in _validate_
    self._check_n_records(record_count_map[level])
  File "/home/qiime2/miniconda/envs/qiime2-2017.12/lib/python3.5/site-packages/q2_types/per_sample_sequences/_format.py", line 130, in _check_n_records
    % (i * 4 + 1))
qiime2.plugin.model.base.ValidationError: Missing sequence for record beginning on line 5

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/qiime2/miniconda/envs/qiime2-2017.12/lib/python3.5/site-packages/q2cli/commands.py", line 224, in __call__
    results = action(**arguments)
  File "<decorator-gen-356>", line 2, in trim_single
  File "/home/qiime2/miniconda/envs/qiime2-2017.12/lib/python3.5/site-packages/qiime2/sdk/action.py", line 228, in bound_callable
    output_types, provenance)
  File "/home/qiime2/miniconda/envs/qiime2-2017.12/lib/python3.5/site-packages/qiime2/sdk/action.py", line 391, in _callable_executor_
    spec.qiime_type, output_view, spec.view_type, prov)
  File "/home/qiime2/miniconda/envs/qiime2-2017.12/lib/python3.5/site-packages/qiime2/sdk/result.py", line 239, in _from_view
    result = transformation(view)
  File "/home/qiime2/miniconda/envs/qiime2-2017.12/lib/python3.5/site-packages/qiime2/core/transform.py", line 57, in transformation
    self.validate(view)
  File "/home/qiime2/miniconda/envs/qiime2-2017.12/lib/python3.5/site-packages/qiime2/core/transform.py", line 131, in validate
    view.validate('min')
  File "/home/qiime2/miniconda/envs/qiime2-2017.12/lib/python3.5/site-packages/qiime2/plugin/model/directory_format.py", line 171, in validate
    getattr(self, field)._validate_members(collected_paths, level)
  File "/home/qiime2/miniconda/envs/qiime2-2017.12/lib/python3.5/site-packages/qiime2/plugin/model/directory_format.py", line 101, in _validate_members
    self.format(path, mode='r').validate(level)
  File "/home/qiime2/miniconda/envs/qiime2-2017.12/lib/python3.5/site-packages/qiime2/plugin/model/file_format.py", line 29, in validate
    ) from e
qiime2.plugin.model.base.ValidationError: /tmp/q2-CasavaOneEightSingleLanePerSampleDirFmt-fxl7g674/MC1_run1_Teleo_0_L001_R1_001.fastq.gz is not a(n) FastqGzFormat file:

Missing sequence for record beginning on line 5

Plugin error from cutadapt:

  /tmp/q2-CasavaOneEightSingleLanePerSampleDirFmt-fxl7g674/MC1_run1_Teleo_0_L001_R1_001.fastq.gz is not a(n) FastqGzFormat file:

  Missing sequence for record beginning on line 5

See above for debug info.

Any suggestions on how I can get this to produce a trimmed output file? It looks like it’s able to trim the primers and then something goes wrong.

Thanks!

Erin

thermokarst · March 8, 2018, 2:50pm

Hey @Erin_dfo! Sorry about this - I think it might actually be a bug, but I can’t be certain yet. Would you be able to share your data with me (a link to download it from Dropbox or Google Drive, for example, shared here or in a direct message)? That would let me reproduce this issue locally, then I can follow up on my hunch about this bug (does that make it a bug hunch? that sounds like a great name for a band!). Thanks for your cooperation!

Erin_dfo · March 8, 2018, 10:35pm

Hi Matthew! Thanks for the quick reply. My organization has a bunch of rules about sharing data, so I don’t think I can get the files onto Dropbox or Google Drive very easily. Would there be any information I could pull from the files that would be helpful to you?

For what it’s worth, I am running Qiime2 in VirtualBox on a Windows machine. After getting these errors in the 2017.12 version, I updated to 2018.2 today, but I still get the same error.

thermokarst · March 8, 2018, 10:38pm

If you have another mechanism to share, by all means, but I understand if you can't

Unfortunately no, we need the reads themselves to diagnose.

Thanks for the info - I think the issue has to do with a mismatch in data formatting expectations between cutadapt and qiime2, so I don't think the environment you run in will matter here. This is still just a hunch, which is why I am trying to drum up data to reproduce the error with. I have another lead that I am following up on right now, too. Thanks!

Erin_dfo · March 8, 2018, 10:39pm

This is what I get from the head of the fastq files, if that’s of any use:

@KBM1K:00004:00046
CTTCCGGTACACTTACCATGTTACGAC
+
6;@C6;06;;;;;8//)./--)---55
@KBM1K:00005:00004
CTTCCGGTACACTTACCATGTTACGACTTGCCTCCCCTTTGTTCGGTTAACTTCTTAGTTAGAAGGGTAGATTGAACTTGGGGAGAGTGACGGGCGGTGT
+
CC@C@C@CCCCC>=@@;>.11..-57.;<>>@BCDC;CD?CCACCACACACC@CCACDC@CBC@CC>CCDCDADD:>>ADDF:DE??@CCAAA0;;6;C:
@KBM1K:00005:00022
CTTCCGGTACACTTACCATGTTACGACTTGCCTCCCCTTTGTTCGGTTAACTTCTTAGTTAGAAGGGTAGATTGAACTTGGGGAGAGTGACAGCTTACGTTACGTTCTTGAGCATTCGATCGATGTACAGCTACGTACGTTCTGAGCATCGATCGATGTACAGCTACGTACGTCTGAGCATCGATCGATGTACAGCTACGTACGTCTGAGCATCGATCGATGTACAGCTACGTACGTCTGAGCATCGATCGATGTACAGCTACGTACGTCTGAGCATCATCGATGTACAGCTAGTACGTCTTGAGCATC

thermokarst · March 8, 2018, 10:43pm

Thanks @Erin_dfo - I guess you could run head -n 100 on Teleo_1_L001_R1_001.fastq.gz, and provide that here. Just to clarify (I was re-reading the title of this topic) - the error message you are seeing has nothing to do with the input data, it has to do with errors in the data that it is trying to write out, after processing through cutadapt (so, its not that cutadapt thinks your fastq.gz files aren’t fastq.gz, but rather that QIIME 2 doesn’t think the output fastq.gz files from cutadapt are fastq.gz files - yikes!)

Erin_dfo · March 9, 2018, 3:19pm

Hi Matthew,

Not sure if this is relevant, but when I enter

head -n 100 MC1_run1_Teleo_0_L001_R1_001.fastq.gz

I get the error message

head: cannot open 'MC1_run1_Teleo_0_L001_R1_001.fastq.gz' for reading: No such file or directory

If I enter it as head -n 100 MC1_run1_Teleo_0_L001_R1_001.fastq instead, this is the output

@KBM1K:00004:00046
CTTCCGGTACACTTACCATGTTACGAC
+
6;@C6;06;;;;;8//)./--)---55
@KBM1K:00005:00004
CTTCCGGTACACTTACCATGTTACGACTTGCCTCCCCTTTGTTCGGTTAACTTCTTAGTTAGAAGGGTAGATTGAACTTGGGGAGAGTGACGGGCGGTGT
+
CC@C@C@CCCCC>=@@;>.11..-57.;<>>@BCDC;CD?CCACCACACACC@CCACDC@CBC@CC>CCDCDADD:>>ADDF:DE??@CCAAA0;;6;C:
@KBM1K:00005:00022
CTTCCGGTACACTTACCATGTTACGACTTGCCTCCCCTTTGTTCGGTTAACTTCTTAGTTAGAAGGGTAGATTGAACTTGGGGAGAGTGACAGCTTACGTTACGTTCTTGAGCATTCGATCGATGTACAGCTACGTACGTTCTGAGCATCGATCGATGTACAGCTACGTACGTCTGAGCATCGATCGATGTACAGCTACGTACGTCTGAGCATCGATCGATGTACAGCTACGTACGTCTGAGCATCGATCGATGTACAGCTACGTACGTCTGAGCATCATCGATGTACAGCTAGTACGTCTTGAGCATC
+
;;7;<;@DD>?@@?;;@C<<?@CAAC<;6?>@@CDF;CC>CCADCACACADC>??BDCCACCC;>?<?DDEE>?>2;;6;;;44@@@BCB>??B7BC>?<=C44.4:.2229@C7C>;;BAA@::22222----.-:77'---:::-0-;2;:@6;6??BBB6666;@288;0.04------6684004>>=1111441>>>;66;;?>?@>444411844>>>>@;;6?AA:?;:4:3:46?AAB;;?A;66A;-----/-/---.--78:-0-@@@BBAB@B;;6?AAA4<44;;60/'-------
@KBM1K:00005:00030
CTTCCGGTACACTTACCATGTTACGACTTGCCTCCCCTTTGTTCGGTTAACTTCTTAGTTAGAAGGGTAGATTGAACTTGGGGAGAGTGACGGGCGGTGT
+
??AC@B@C>>>BC@AA?B<<<7<66;/-....-.7767@;CCACCACADA>?<>@>@CCACCCACC?CCCCC>??=>>=@C@7>>;;;C:;;B:;;>A;:
@KBM1K:00007:00023
CTTCCGGTACACTTACCATGTTACGACTTGCCTCCCCTTTGTTCGGTTAACTTCTTAGTTAGAAGGGTAGATTGAACTTGGGGAGAGTGACGGGCGGTGTATCACCGATGTACAGCTACGTACGTCTGAGCATCGATCGATGTACAGCTACGTACGTCTGAGCATCGATCGATGTACAGCTACGTACGTCTGAGCATCGATCGATGTACAGCTACGTACGTCTGAGCATCGATCGATGTACAGCTACGTACGTCTGAGCATCGATCGATGTACAGCTACGTACGTCTGAGCATCGATCG
+
CCACACACCCCD===767.//.--55-;06=06?DD;DD?CCADDACBCA??>?CADCCA?@?BCC?CCCCCACC=??;?DC:CC???CCCCE<>;4;;::;:::4:92::;42222222@C258C8:2C@::222225?C=@C=@@?;:4;?::?@@@@><<66144BBBB?A;;6;??B;??@@A=;6;@=8818441@@B;;????A=;;6??B;66=;;;CBBA?;;6BBB4432990--0-:94:4.--.1<=?8711//-626760--40--;;;444--.62.--0-27272
@KBM1K:00007:00033
ACACCGCCCGTCACTCTCCCCAAGTTCAATCTACCCTTCTAACTAAGAAGTTAACCGTAACAAAGGGAGGCAATGTCG
+
;;;;7;;;1;B;;;AB;;;;,6666:CA0<6<666,606660;.-36<..3535.---36<<--)54.49.4444=
@KBM1K:00007:00043
ACACCGCCCGTCACTCTCCCCAAGTTCAATCTACCCTTCTAACTAAGAAGTTAACCGAACAAAGGGGAGGCAAGTCGTAACATGGTAAGTGTACCGGAAG
+
?@DD<?@E=>?DCDDDCCHH;D:<<6<CC?>?AC=>9:
@KBM1K:00008:00019
ACACCGCCCGTCACTCTCCCCAAGTTCAATCTACCCTTCTAACTAAGAAGTTAACCGAACAAAGGGGAGGCAAGTCGTAACATGGTAAGTGTACCGGAAG
+
CD>><>;D1>?CCCCCCCDC;C?@@ACC>EA<;;;1;766=6@BC<??=@A7;6;06;7;;B1;;;/;;.4969C>;>?>1/;<80/+.--4:C>C4:5:
@KBM1K:00008:00022
ACACCGCCCGTCACTCTCCCCAAGTTCAATCTACCCTTCTAACTAAGAAGTTAACCGAACAAAGGGGAGGCAAGTCGTAACATGGTAAGTGTACCGGAAG
+
CC??>?CC<@@CDCCCCCEE:C>???E@ECCCCCCACCDACCC<??<==;A;A06;05;;1;CC/;474;.44;44?@CCCDA??>?CCCCC@C@C<:
@KBM1K:00008:00023
CTTCCGGTACACTTACCATGTTACGACTTGCCTCCCCTTTGTTCGGTTAACTTCTTAGTTAGAAGGGTAGATTGAACTTGGGGAGAGTGACGGGCGGTGT
+
CDACADA@@@CC@=;:8;./.)--55-;06606;CC:CD=@?@CCCEAC@?>7;@>@CCACCCACC>CCCCC@CC=?>=@@@.;;>>?CCCAA9;//00
@KBM1K:00009:00041
CTTCCGGTACACTTACCATGTTACGACTTGCCTCCCCTTTGTTCGGTTAACTTCTTAGTTAGAAGGGTAGATTGAACTTGGGGAGAGTGACGGGCGGTGT
+
DDACACACDCC>....8;/0/.-----/+--66;67....?@BBC??=>A?@@@DACCC@>>>=@C>CBB?>;;B;>?ACDD8@?CCC>>7;=./08;00
@KBM1K:00010:00022
CTTCCGGTACACTTACCATGTTACGACTTGCCTCCCCTTTGCAATTATTAGTTTTTAGTTAATTGTATTATTAGGCTCGGGGAGAGTGACGGGCGGTGT
+

6;=@=;;;;;0+0<8//--3---5-/)--35567/5;+;;;7;7>>@CDDFCC5;0/+0+0+--553??>?C>???=ED7;::<:/--/<*--)-3-
@KBM1K:00010:00042
ACACCGCCCGTCACTCTCCCCTGTTAAATTGCGTCAACTGTAATTAACCTGAAAGCACCAACGAGGGGAGGCAAGTCGTAACATGGTAAGTGTACCGGAAG

CDCDADEE>@@DCCCC=66;/<<<7<A6A7<<<;?C@ABD;>=@@C=@?B;;;1;;;;=@7;;;;AA/5>@CC;??CCCCA??@CADC@B???CBAD>@>?
@KBM1K:00010:00046
ACACCGCCCGTCACTCTCCCCAAGTTCAATCTA
+
??;C7;;;166====C>;;;/;0<<0--)-50.
@KBM1K:00011:00035
CTTCCGGTACACTTACCATGTTACGACTTG
+
CC@CACBCCCB>.../8;/--36666-.)-
@KBM1K:00012:00007
ACACCGCCCGTCACTCTCCCCAAGTTCAATCTACCCTTCTAACTAAGAAGTTAACCGAACAAAGGGGAGGCAAGTCGTAACATGGTAAGTGTACCGGAAG
+
DDDCACCC=@@DDCCCCCCC;D:>?ACF@@@CECC;;7;6=:CCC@CC???ADAE@CDC@??<?DD8CC?CB6>?CCCCACDCCACCACDCCDD>?>??@
@KBM1K:00012:00029
CTTCCGGTACACTTACCATGTTACGACTTGCCTCCCCTTTGTTCGGTTAACTTCTTAGTTAGAAGGGTAGATTGAACTTGGGGAGAGTGACGGGCGGTGT
+
CCBCAC@CCCCCC@AC?A<<<66DDB<606606;BB2BB;??>@@ACACA@?=>;>@;><;;;7>>:>C@@@;;;6??@CFE8???CCCBB@C:;;6;;:
@KBM1K:00013:00026
CTTCCGGTACACTTACCATGTTACGACTTGCCTCCCCTTTGTTCGGTTAACTTCTTAGTTAGAAGGGTAGATTGAACTTGGGGAGAGTGACGGGCGGTGT
+
CC6;6;7;@>@@>=>>=D.0.67556-.)--)-.776=;;CDA??>;>A7;;7;606AB=;;;7;;,;;;;CACC;;;6;;;.4@;;;@9;;;04;>B;:
@KBM1K:00013:00032
TCCGGTACACTTACCATGTTACGACTTGCCTCCCCTCGTTGGTGCTTTCAGGTTAATTACAGTTGACGCAATTTTAACAGGGGAGAGTGACGGGCAGTGT
+
6@7;7;;;;ABA;;8;/--)..@6;;6;;?ABCB2AA;;6;>;<<;;1;;@=@?B7;6;;;;<7<ABA;.)///(.+-----%-===B=>;;;1;;;;B;
@KBM1K:00013:00043
CTTCCGGTACACTTACCATGTTACGACTTGCCTCCCCTTTGTTCGGTTAACTTCTTAGTTAGAAGGGTAGATTGAACTTGGGGAGAGTGACGGGCGGTGT
+
;><;BDACD@@@CABB6<<<<7<A==<;666066AA:CE>AAA;;0;:A6;;7;;06@@@;;;0;;,5@@@@;>;6;;6;;;444@@;:;;B0;;>;;:
@KBM1K:00015:00007
CTTCCGGTACACTTACCATGTTACGACTTGCCTCCCCTTTGTTCGGTTAACTTCTTAGTTAGAAGGGTAGATTGAACTTGGGGAGAGTGACGGGCGGTGT
+
CC@CACACCCD@@@?C@D@@@B@??D@?7;@?;;;;,66,6B?CCAC@C@CCACC;ACDAC;;7;;1;CC;42;@<;;4;;;494;;CC;;;0;;=BCC
@KBM1K:00017:00043
CTTCCGGTACACTTACCATGTTACGACTTGCCTCCCCTCGTTGGTGCTTTCAGGTTAATTACAGTTGACGCAATTTAACAGGGGAGAGTGACGGGCGGTGT
+
@CACACADCB@@8)008;0--366688:--35;67/7;>;7;0;;>?C?C<?7<>A7<7;AA;;7;;;;;A7<<1;7;BCCC;EECCCCCEDD?@?<>@BCCD;CC>CCACC@CACA??=?CACDCACCC>??<?CCCC@CC<@@4;;;.;;;;;CA;;;0;;?CCC @KBM1K:00019:00013 ACACCGCCCGTCACTCTCCCCAAGTTCAATCTACCCTTCTAACTAAGAAGTTAACCGAACAAAGGGGAGGCAAGTCGTAACATGGTAAGTGTACCGGAAG + @@;><;BC=CCCC???CCCC:EADCACEACCCA==1;06CC0>?C@CC@CCAC@C<>>A>?>:>DD8??ACDACCC?>>;;AAC=?>?>>>?=@=A<B9:
@KBM1K:00020:00023
CTTCCGGTACACTTACCATGTTACGACTTGCCTCCCCTTTTTGATTATTAGGTTTTAGTTAATTGATTGGGGTTTTGGGGAGAGTGACGGGCGGTGGT
+
CC=?>>@@@;>>C@BCAC>>@?@?DEDC=?;7;;;;/;;;;,;BB;//)//)/66%..7=>=>=@??7;;;/;;5/5??15>>CC;;;;;044.44.4
(qiime2-2018.2) qiime2@qiime2core2018-2:~/Documents/MC1_run1$ head -n 100 MC1_run1_Teleo_0_L001_R1_001.fastq.gz
head: cannot open 'MC1_run1_Teleo_0_L001_R1_001.fastq.gz' for reading: No such file or directory
(qiime2-2018.2) qiime2@qiime2core2018-2:~/Documents/MC1_run1$ head -n 100 MC1_run1_Teleo_0_L001_R1_001.fastq
@KBM1K:00004:00046
CTTCCGGTACACTTACCATGTTACGAC
+
6;@C6;06;;;;;8//)./--)---55
@KBM1K:00005:00004
CTTCCGGTACACTTACCATGTTACGACTTGCCTCCCCTTTGTTCGGTTAACTTCTTAGTTAGAAGGGTAGATTGAACTTGGGGAGAGTGACGGGCGGTGT
+
CC@C@C@CCCCC>=@@;>.11..-57.;<>>@BCDC;CD?CCACCACACACC@CCACDC@CBC@CC>CCDCDADD:>>ADDF:DE??@CCAAA0;;6;C:
@KBM1K:00005:00022
CTTCCGGTACACTTACCATGTTACGACTTGCCTCCCCTTTGTTCGGTTAACTTCTTAGTTAGAAGGGTAGATTGAACTTGGGGAGAGTGACAGCTTACGTTACGTTCTTGAGCATTCGATCGATGTACAGCTACGTACGTTCTGAGCATCGATCGATGTACAGCTACGTACGTCTGAGCATCGATCGATGTACAGCTACGTACGTCTGAGCATCGATCGATGTACAGCTACGTACGTCTGAGCATCGATCGATGTACAGCTACGTACGTCTGAGCATCATCGATGTACAGCTAGTACGTCTTGAGCATC
+
;;7;<;@DD>?@@?;;@C<<?@CAAC<;6?>@@CDF;CC>CCADCACACADC>??BDCCACCC;>?<?DDEE>?>2;;6;;;44@@@BCB>??B7BC>?<=C44.4:.2229@C7C>;;BAA@::22222----.-:77'---:::-0-;2;:@6;6??BBB6666;@288;0.04------6684004>>=1111441>>>;66;;?>?@>444411844>>>>@;;6?AA:?;:4:3:46?AAB;;?A;66A;-----/-/---.--78:-0-@@@BBAB@B;;6?AAA4<44;;60/'-------
@KBM1K:00005:00030
CTTCCGGTACACTTACCATGTTACGACTTGCCTCCCCTTTGTTCGGTTAACTTCTTAGTTAGAAGGGTAGATTGAACTTGGGGAGAGTGACGGGCGGTGT
+
??AC@B@C>>>BC@AA?B<<<7<66;/-....-.7767@;CCACCACADA>?<>@>@CCACCCACC?CCCCC>??=>>=@C@7>>;;;C:;;B:;;>A;:
@KBM1K:00007:00023
CTTCCGGTACACTTACCATGTTACGACTTGCCTCCCCTTTGTTCGGTTAACTTCTTAGTTAGAAGGGTAGATTGAACTTGGGGAGAGTGACGGGCGGTGTATCACCGATGTACAGCTACGTACGTCTGAGCATCGATCGATGTACAGCTACGTACGTCTGAGCATCGATCGATGTACAGCTACGTACGTCTGAGCATCGATCGATGTACAGCTACGTACGTCTGAGCATCGATCGATGTACAGCTACGTACGTCTGAGCATCGATCGATGTACAGCTACGTACGTCTGAGCATCGATCG
+
CCACACACCCCD===767.//.--55-;06=06?DD;DD?CCADDACBCA??>?CADCCA?@?BCC?CCCCCACC=??;?DC:CC???CCCCE<>;4;;::;:::4:92::;42222222@C258C8:2C@::222225?C=@C=@@?;:4;?::?@@@@><<66144BBBB?A;;6;??B;??@@A=;6;@=8818441@@B;;????A=;;6??B;66=;;;CBBA?;;6BBB4432990--0-:94:4.--.1<=?8711//-626760--40--;;;444--.62.--0-27272
@KBM1K:00007:00033
ACACCGCCCGTCACTCTCCCCAAGTTCAATCTACCCTTCTAACTAAGAAGTTAACCGTAACAAAGGGAGGCAATGTCG
+
;;;;7;;;1;B;;;AB;;;;,6666:CA0<6<666,606660;.-36<..3535.---36<<--)54.49.4444=
@KBM1K:00007:00043
ACACCGCCCGTCACTCTCCCCAAGTTCAATCTACCCTTCTAACTAAGAAGTTAACCGAACAAAGGGGAGGCAAGTCGTAACATGGTAAGTGTACCGGAAG
+
?@DD<?@E=>?DCDDDCCHH;D:<<6<CC?>?AC=>9:
@KBM1K:00008:00019
ACACCGCCCGTCACTCTCCCCAAGTTCAATCTACCCTTCTAACTAAGAAGTTAACCGAACAAAGGGGAGGCAAGTCGTAACATGGTAAGTGTACCGGAAG
+
CD>><>;D1>?CCCCCCCDC;C?@@ACC>EA<;;;1;766=6@BC<??=@A7;6;06;7;;B1;;;/;;.4969C>;>?>1/;<80/+.--4:C>C4:5:
@KBM1K:00008:00022
ACACCGCCCGTCACTCTCCCCAAGTTCAATCTACCCTTCTAACTAAGAAGTTAACCGAACAAAGGGGAGGCAAGTCGTAACATGGTAAGTGTACCGGAAG
+
CC??>?CC<@@CDCCCCCEE:C>???E@ECCCCCCACCDACCC<??<==;A;A06;05;;1;CC/;474;.44;44?@CCCDA??>?CCCCC@C@C<:
@KBM1K:00008:00023
CTTCCGGTACACTTACCATGTTACGACTTGCCTCCCCTTTGTTCGGTTAACTTCTTAGTTAGAAGGGTAGATTGAACTTGGGGAGAGTGACGGGCGGTGT
+
CDACADA@@@CC@=;:8;./.)--55-;06606;CC:CD=@?@CCCEAC@?>7;@>@CCACCCACC>CCCCC@CC=?>=@@@.;;>>?CCCAA9;/*/00
@KBM1K:00009:00041
CTTCCGGTACACTTACCATGTTACGACTTGCCTCCCCTTTGTTCGGTTAACTTCTTAGTTAGAAGGGTAGATTGAACTTGGGGAGAGTGACGGGCGGTGT
+
DDACACACDCC>....8;/0/.-----/+--66;67....?@BBC??=>A?@@@DACCC@>>>=@C>CBB?>;;B;>?ACDD8@?CCC>>7;=./08;00
@KBM1K:00010:00022
CTTCCGGTACACTTACCATGTTACGACTTGCCTCCCCTTTGCAATTATTAGTTTTTAGTTAATTGTATTATTAGGCTCGGGGAGAGTGACGGGCGGTGT
+

6;=@=;;;;;0+0<8//--3---5-/)--35567/5;+;;;7;7>>@CDDFCC5;0/+0+0+--553??>?C>???=ED7;::<:/--/<*--)-3-
@KBM1K:00010:00042
ACACCGCCCGTCACTCTCCCCTGTTAAATTGCGTCAACTGTAATTAACCTGAAAGCACCAACGAGGGGAGGCAAGTCGTAACATGGTAAGTGTACCGGAAG

CDCDADEE>@@DCCCC=66;/<<<7<A6A7<<<;?C@ABD;>=@@C=@?B;;;1;;;;=@7;;;;AA/5>@CC;??CCCCA??@CADC@B???CBAD>@>?
@KBM1K:00010:00046
ACACCGCCCGTCACTCTCCCCAAGTTCAATCTA
+
??;C7;;;166====C>;;;/;0<<0--)-50.
@KBM1K:00011:00035
CTTCCGGTACACTTACCATGTTACGACTTG
+
CC@CACBCCCB>.../8;/--36666-.)-
@KBM1K:00012:00007
ACACCGCCCGTCACTCTCCCCAAGTTCAATCTACCCTTCTAACTAAGAAGTTAACCGAACAAAGGGGAGGCAAGTCGTAACATGGTAAGTGTACCGGAAG
+
DDDCACCC=@@DDCCCCCCC;D:>?ACF@@@CECC;;7;6=:CCC@CC???ADAE@CDC@??<?DD8CC?CB6>?CCCCACDCCACCACDCCDD>?>??@
@KBM1K:00012:00029
CTTCCGGTACACTTACCATGTTACGACTTGCCTCCCCTTTGTTCGGTTAACTTCTTAGTTAGAAGGGTAGATTGAACTTGGGGAGAGTGACGGGCGGTGT
+
CCBCAC@CCCCCC@AC?A<<<66DDB<606606;BB2BB;??>@@ACACA@?=>;>@;><;;;7>>:>C@@@;;;6??@CFE8???CCCBB@C:;;6;;:
@KBM1K:00013:00026
CTTCCGGTACACTTACCATGTTACGACTTGCCTCCCCTTTGTTCGGTTAACTTCTTAGTTAGAAGGGTAGATTGAACTTGGGGAGAGTGACGGGCGGTGT
+
CC6;6;7;@>@@>=>>=D.0.67556-.)--)-.776=;;CDA??>;>A7;;7;606AB=;;;7;;,;;;;CACC;;;6;;;.4@;;;@9;;;04;>B;:
@KBM1K:00013:00032
TCCGGTACACTTACCATGTTACGACTTGCCTCCCCTCGTTGGTGCTTTCAGGTTAATTACAGTTGACGCAATTTTAACAGGGGAGAGTGACGGGCAGTGT
+
6@7;7;;;;ABA;;8;/--)..@6;;6;;?ABCB2AA;;6;>;<<;;1;;@=@?B7;6;;;;<7<ABA;.)///(.+-----%-===B=>;;;1;;;;B;
@KBM1K:00013:00043
CTTCCGGTACACTTACCATGTTACGACTTGCCTCCCCTTTGTTCGGTTAACTTCTTAGTTAGAAGGGTAGATTGAACTTGGGGAGAGTGACGGGCGGTGT
+
;><;BDACD@@@CABB6<<<<7<A==<;666066AA:CE>AAA;;0;:A6;;7;;06@@@;;;0;;,5@@@@;>;6;;6;;;*444@@;:;;B0;;>;;:
@KBM1K:00015:00007
CTTCCGGTACACTTACCATGTTACGACTTGCCTCCCCTTTGTTCGGTTAACTTCTTAGTTAGAAGGGTAGATTGAACTTGGGGAGAGTGACGGGCGGTGT
+
CC@CACACCCD@@@?C@D@@@B@??D@?7;@?;;;;,66,6B?CCAC@C@CCACC;ACDAC;;7;;1;CC;42;@<;;4;;;494;;CC;;;0;;=BCC
@KBM1K:00017:00043
CTTCCGGTACACTTACCATGTTACGACTTGCCTCCCCTCGTTGGTGCTTTCAGGTTAATTACAGTTGACGCAATTTAACAGGGGAGAGTGACGGGCGGTGT
+
@CACACADCB@@8)008;0--366688:--35;67/7;>;7;0;;>?C?C<?7<>A7<7;AA;;7;;;;;A7<<1;7;BCCC;EECCCCCEDD?@?<>@BCCD;CC>CCACC@CACA??=?CACDCACCC>??<?CCCC@CC<@@4;;;.;;;;;CA;;;0;;?CCC @KBM1K:00019:00013 ACACCGCCCGTCACTCTCCCCAAGTTCAATCTACCCTTCTAACTAAGAAGTTAACCGAACAAAGGGGAGGCAAGTCGTAACATGGTAAGTGTACCGGAAG + @@;><;BC=CCCC???CCCC:EADCACEACCCA==1;06CC0>?C@CC@CCAC@C<>>A>?>:>DD8??ACDACCC?>>;;AAC=?>?>>>?=@=A<B9:
@KBM1K:00020:00023
CTTCCGGTACACTTACCATGTTACGACTTGCCTCCCCTTTTTGATTATTAGGTTTTAGTTAATTGATTGGGGTTTTGGGGAGAGTGACGGGCGGTGGT
+
CC=?>>@@@;>>C@BCAC>>@?@?DEDC=?;7;;;;/;;;;,;BB;//)//)/66%..7=>=>=@??7;;;/;;5/5??15>>CC;;;;;044.44.4

Hope that is helpful! I will keep an eye out for the bug fix.

thermokarst · March 9, 2018, 10:39pm

Woohoo!!! !!!

With that little bit of data you provided, I was able to reproduce the error. Not only that, but I was able to see () exactly what the issue is. Running the same command you provided above, but instead of using q2-cutadapt we go through cutadapt directly, I get some reads out of cutadapt that look like this:

Yowzah! So, hopefully that QIIME 2 error makes a bit more sense now --- it was complaining about records that had no sequence associated with it.

I have opened up an issue on the relevant bug tracker for us to fix this up. In the meantime, you could export your demuxed reads and run cutadapt on its own, then reimport your trimmed reads, just make sure to include the flag -m 1 to specify a minimum read length (you could use a larger value, but don't set it any lower than this. For example:

cutadapt \
  --cores 1 \
  --error-rate 0.1 \
  --times 1 \
  --overlap 3 \
  -o trimmed/Teleo_1_L001_R1_001.fastq.gz \
  --adapter CTTCCGGTACACTTACCATG \
  --front ACACCGCCCGTCACTCT \
  -m 1 \
  input/Teleo_1_L001_R1_001.fastq.gz

Thanks so much for your help with identifying this bug - you saved the day!

Erin_dfo · March 12, 2018, 1:35pm

Hi Matthew,

I’m glad that that little bit of data was helpful! I will keep an eye out for the bug fix.

In the meantime, I had success with a) importing just one .fastq file at a time, instead of several in one fastqmanifest, and b) switching to the linked adapters setting (which I was going to try eventually, but had figured wouldn’t work after the original settings glitched).

The command that worked for me was:

qiime cutadapt trim-single --i-demultiplexed-sequences MC1_run1_Teleo.qza --p-front ACACCGCCCGTCACTCT…CATGGTAAGTGTACCGGAAG --p-front CTTCCGGTACACTTACCATG…AGAGTGACGGGCGGTGT --p-error-rate 0.1 --o-trimmed-sequences MC1_run1_Teleo_trimmed_10error.qza --verbose

I should note that when I played around with different error rates, I got the same bug for any error rates between 0.25 and 0.35. I tried a bunch of different rates in this range, and the glitch was consistent. But lower error rates worked for me once I corrected some mistakes on my part, so I have workable data for now.

Thanks for your help!

thermokarst · March 12, 2018, 1:47pm

Good to hear! Just to clarify, my suggestion above was to run cutadapt outside of QIIME 2, that way you can use the -m 1 flag, which enforces a minimum sequence length. You can just run cutadapt commands as-is within your QIIME 2 environment, because we bundle cutadapt for use in q2-cutadapt.

It sounds like you were able to come up with a solution by tweaking the error-rate parameter, but personally I would be a bit wary of this, because now you have applied different error rate settings to one of your samples, which might be hard to justify in a methods section of a paper. Personally I think saying that only non-zero length sequences were considered (i.e. -m 1) in your analysis, across all your samples, might be a bit easier to reason about. Thanks!

system · April 12, 2018, 7:47pm

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.