metadata newlines don't work (Mac OS X, cutadapt)

[wasteoftime]

First (and last?) time running qiime. All I am trying to do at the moment is demultiplex. Qiime cutadapt is unable parse my metadata file no matter what I do to it. I've spent hours trying to fix this ridiculous bug. It seems most likely that the cutadapt plugin isn't able to parse newlines on Macs when reading the barcodes even though somehow it IS able to determine when I have non-unique values in the ID column (which I had to fix before getting to this error). Shouldn't qiime be able to use TSV files generated by pandas?

Command
qiime cutadapt demux-paired --i-seqs multiplexed-seqs.qza --m-forward-barcodes-file barcodes.txt --m-forward-barcodes-column barcode --m-reverse-barcodes-file barcodes.txt --m-reverse-barcodes-column barcode --o-per-sample-sequences demultiplexed-seqs.qza --o-untrimmed-sequences untrimmed.qza --verbose

Output
/opt/anaconda3/envs/qiime2-2023.2/lib/python3.8/site-packages/q2_cutadapt/_demux.py:108: FutureWarning: iteritems is deprecated and will be removed in a future version. Use .items instead.
for (sample_id, barcode) in barcode_series.iteritems():
Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

Command: cutadapt --front file:/var/folders/f8/9qkks6g173gdzqhzpvsksrlxs16y_b/T/tmpd258tay5 --error-rate 0.1 --minimum-length 1 -o /var/folders/f8/9qkks6g173gdzqhzpvsksrlxs16y_b/T/q2-CasavaOneEightSingleLanePerSampleDirFmt-z6iysql6/{name}.1.fastq.gz --untrimmed-output /var/folders/f8/9qkks6g173gdzqhzpvsksrlxs16y_b/T/q2-MultiplexedPairedEndBarcodeInSequenceDirFmt-m3cjkbvr/forward.fastq.gz --pair-adapters -G file:/var/folders/f8/9qkks6g173gdzqhzpvsksrlxs16y_b/T/tmpzv6ex6v6 -p /var/folders/f8/9qkks6g173gdzqhzpvsksrlxs16y_b/T/q2-CasavaOneEightSingleLanePerSampleDirFmt-z6iysql6/{name}.2.fastq.gz --untrimmed-paired-output /var/folders/f8/9qkks6g173gdzqhzpvsksrlxs16y_b/T/q2-MultiplexedPairedEndBarcodeInSequenceDirFmt-m3cjkbvr/reverse.fastq.gz /var/folders/f8/9qkks6g173gdzqhzpvsksrlxs16y_b/T/qiime
2/xxx/data/8e113155-4129-4de0-88c5-a07ef447d08e/data/forward.fastq.gz /var/folders/f8/9qkks6g173gdzqhzpvsksrlxs16y_b/T/qiime2/xxx/data/8e113155-4129-4de0-88
c5-a07ef447d08e/data/reverse.fastq.gz -j 1

This is cutadapt 4.2 with Python 3.8.16
Command line parameters: --front file:/var/folders/f8/9qkks6g173gdzqhzpvsksrlxs16y_b/T/tmpd258tay5 --error-rate 0.1 --minimum-length 1 -o /var/folders/f8/9qkks6g173gdzqhzpvsksrlxs16y_b/T/q2-CasavaOneEightSingleLanePerSampleDirFmt-z6iysql6/{name}.1.fastq.gz --untrimmed-output /var/folders/f8/9qkks6g173gdzqhzpvsksrlxs16y_b/T/q2-MultiplexedPairedEndBarcodeInSequenceDirFmt-m3cjkbvr/forward.fastq.gz --pair-adapters -G file:/var/folders/f8/9qkks6g173gdzqhzpvsksrlxs16y_b/T/tmpzv6ex6v6 -p /var/folders/f8/9qkks6g173gdzqhzpvsksrlxs16y_b/T/q2-CasavaOneEightSingleLanePerSampleDirFmt-z6iysql6/{name}.2.fastq.gz --untrimmed-paired-output /var/folders/f8/9qkks6g173gdzqhzpvsksrlxs16y_b/T/q2-MultiplexedPairedEndBarcodeInSequenceDirFmt-m3cjkbvr/reverse.fastq.gz /var/folders/f8/9qkks6g173gdzqhzpvsksrlxs16y_b/T/qiime2/xxx/data/8e113155-4129-4de0-88c5-a07ef447d08e/data/forward.fastq.gz /var/folders/f8/9qkks6g173gdzqhzpvsksrlxs16y_b/T/qiime2/xxx/data/8e113155-4129-4de0-88c5-a07ef447d08e/data/reverse.fastq.gz -j 1
ERROR: Character '2' in adapter sequence 'ND2238B3 ACCTGATCCGCA2238B5 GTTGACCATCGC2238G11 TGACTGCGTTAG2238B9 CATACCGTGAGT2238D2 CCGAGTACAATCPCRNEG1 AGCCTGGTACCT2238E12 GAAGACAGCGAC2238F3 TGGACTCAGCTA2238F6 GTCCGATCCTAG2238A3 TCATTAGCGTGG2238B6 TAGGCTCGTGCTPCRZYMO1 CGGTAGTTGATC2416A9 GTACTCGAACCA2416D2 ACTGCTATCGCG2416H4 CACTGAGTACGT2417G1 CCTAAGAGCATC2417B9 ATCGTGTGTTGG2417D1 CGTCCGTATGAA2417C8 GTGCAACCAATC2417D2 ACCAGTGACTCAPCRNEG2 GGATACTCGCATBLANK1 TAGTGCATTCGGBLANK2 AGAACACGGAAG2238B3_V2 GTCCGCAAGTTA2238B5_V2 CCTGTCCTATCT2238G11_V2 TTGTCAGCTGGA2238B9_V2 GATAAGCGCCTT2238D2_V2 ACAGGTAGAGAG2238E12_V2 CTCTATTCCACC2238F3_V2 ACCACGATGCTAPCRNEG3 TGGCGATACGTT2238F6_V2 GCACTGGCATAT2238A3_V2 TGTGCTTGTAGG2238B6_V2 CACAACCACAAC2416A9_V2 AGAGTCTTGCCA2416D2_V2 CTGTGTCCATGG2416H4_V2 TTGCCAAGAGTC2417G1_V2 GGTATGGCTACTPCRZYMO2 CAACACATGCTG2417B9_V2 GCTTGAGCTTGA2417D1_V2 AACATGCATGCC2417C8_V2 TCGGCTTGGAAT2417D2_V2 CTATGAGTCCAG2238B3_V3 ATCTTGGAGTCG2238B5_V3 GAAGGTGAAGGT2238G11_V3 AGTCCACTGGTA2238B9_V3 GCCGATTGTAAC2238D2_V3 CGTAGCCAACAT2238E12_V3 ACGCAATGTCTG2238A3_V3 TAATCGGTGCCA2416A9_V3 CGAGTCACGATTPCRNEG4 AGGTTCTTAGGC2417G1_V3 CAGAATCGCTCA2417C8_V3 TTCGCAGATACG2417D2_V3 ACTCCGATAGACPCRZYMO3 GATAGCACTCGTNAME: BARCODE, DTYPE: OBJECT' is not a valid IUPAC code. Use only characters 'ABCDGHIKMNRSTUVWXY'.
Traceback (most recent call last):
File "/opt/anaconda3/envs/qiime2-2023.2/lib/python3.8/site-packages/q2cli/commands.py", line 352, in call
results = action(**arguments)
File "", line 2, in demux_paired
File "/opt/anaconda3/envs/qiime2-2023.2/lib/python3.8/site-packages/qiime2/sdk/action.py", line 234, in bound_callable
outputs = self.callable_executor(scope, callable_args,
File "/opt/anaconda3/envs/qiime2-2023.2/lib/python3.8/site-packages/qiime2/sdk/action.py", line 381, in callable_executor
output_views = self._callable(**view_args)
File "/opt/anaconda3/envs/qiime2-2023.2/lib/python3.8/site-packages/q2_cutadapt/_demux.py", line 229, in demux_paired
untrimmed = _demux(
File "/opt/anaconda3/envs/qiime2-2023.2/lib/python3.8/site-packages/q2_cutadapt/_demux.py", line 180, in _demux
run_command(cmd)
File "/opt/anaconda3/envs/qiime2-2023.2/lib/python3.8/site-packages/q2_cutadapt/_demux.py", line 37, in run_command
subprocess.run(cmd, check=True)
File "/opt/anaconda3/envs/qiime2-2023.2/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['cutadapt', '--front', 'file:/var/folders/f8/9qkks6g173gdzqhzpvsksrlxs16y_b/T/tmpd258tay5', '--error-rate', '0.1', '--minimum-length', '1', '-o', '/var/folders/f8/9qkks6g173gdzqhzpvsksrlxs16y_b/T/q2-CasavaOneEightSingleLanePerSampleDirFmt-z6iysql6/{name}.1.fastq.gz', '--untrimmed-output', '/var/folders/f8/9qkks6g173gdzqhzpvsksrlxs16y_b/T/q2-MultiplexedPairedEndBarcodeInSequenceDirFmt-m3cjkbvr/forward.fastq.gz', '--pair-adapters', '-G', 'file:/var/folders/f8/9qkks6g173gdzqhzpvsksrlxs16y_b/T/tmpzv6ex6v6', '-p', '/var/folders/f8/9qkks6g173gdzqhzpvsksrlxs16y_b/T/q2-CasavaOneEightSingleLanePerSampleDirFmt-z6iysql6/{name}.2.fastq.gz', '--untrimmed-paired-output', '/var/folders/f8/9qkks6g173gdzqhzpvsksrlxs16y_b/T/q2-MultiplexedPairedEndBarcodeInSequenceDirFmt-m3cjkbvr/reverse.fastq.gz', '/var/folders/f8/9qkks6g173gdzqhzpvsksrlxs16y_b/T/qiime2/xxx/data/8e113155-4129-4de0-88c5-a07ef447d08e/data/forward.fastq.gz', '/var/folders/f8/9qkks6g173gdzqhzpvsksrlxs16y_b/T/qiime2/xxx/data/8e113155-4129-4de0-88c5-a07ef447d08e/data/reverse.fastq.gz', '-j', '1']' returned non-zero exit status 2.

Plugin error from cutadapt:

Command '['cutadapt', '--front', 'file:/var/folders/f8/9qkks6g173gdzqhzpvsksrlxs16y_b/T/tmpd258tay5', '--error-rate', '0.1', '--minimum-length', '1', '-o', '/var/folders/f8/9qkks6g173gdzqhzpvsksrlxs16y_b/T/q2-CasavaOneEightSingleLanePerSampleDirFmt-z6iysql6/{name}.1.fastq.gz', '--untrimmed-output', '/var/folders/f8/9qkks6g173gdzqhzpvsksrlxs16y_b/T/q2-MultiplexedPairedEndBarcodeInSequenceDirFmt-m3cjkbvr/forward.fastq.gz', '--pair-adapters', '-G', 'file:/var/folders/f8/9qkks6g173gdzqhzpvsksrlxs16y_b/T/tmpzv6ex6v6', '-p', '/var/folders/f8/9qkks6g173gdzqhzpvsksrlxs16y_b/T/q2-CasavaOneEightSingleLanePerSampleDirFmt-z6iysql6/{name}.2.fastq.gz', '--untrimmed-paired-output', '/var/folders/f8/9qkks6g173gdzqhzpvsksrlxs16y_b/T/q2-MultiplexedPairedEndBarcodeInSequenceDirFmt-m3cjkbvr/reverse.fastq.gz', '/var/folders/f8/9qkks6g173gdzqhzpvsksrlxs16y_b/T/qiime2/xxx/data/8e113155-4129-4de0-88c5-a07ef447d08e/data/forward.fastq.gz', '/var/folders/f8/9qkks6g173gdzqhzpvsksrlxs16y_b/T/qiime2/xxx/data/8e113155-4129-4de0-88c5-a07ef447d08e/data/reverse.fastq.gz', '-j', '1']' returned non-zero exit status 2.

See above for debug info.

barcodes.txt

id      barcode
2238B3  ACCTGATCCGCA
2238B5  GTTGACCATCGC
2238G11 TGACTGCGTTAG
2238B9  CATACCGTGAGT
2238D2  CCGAGTACAATC
PCRneg1 AGCCTGGTACCT
2238E12 GAAGACAGCGAC
2238F3  TGGACTCAGCTA
2238F6  GTCCGATCCTAG
2238A3  TCATTAGCGTGG
2238B6  TAGGCTCGTGCT
PCRzymo1        CGGTAGTTGATC
2416A9  GTACTCGAACCA
2416D2  ACTGCTATCGCG
2416H4  CACTGAGTACGT
2417G1  CCTAAGAGCATC
2417B9  ATCGTGTGTTGG
2417D1  CGTCCGTATGAA
2417C8  GTGCAACCAATC
2417D2  ACCAGTGACTCA
PCRneg2 GGATACTCGCAT
blank1  TAGTGCATTCGG
blank2  AGAACACGGAAG
2238B3_v2       GTCCGCAAGTTA
2238B5_v2       CCTGTCCTATCT
2238G11_v2      TTGTCAGCTGGA
2238B9_v2       GATAAGCGCCTT
2238D2_v2       ACAGGTAGAGAG
2238E12_v2      CTCTATTCCACC
2238F3_v2       ACCACGATGCTA
PCRneg3 TGGCGATACGTT
2238F6_v2       GCACTGGCATAT
2238A3_v2       TGTGCTTGTAGG
2238B6_v2       CACAACCACAAC
2416A9_v2       AGAGTCTTGCCA
2416D2_v2       CTGTGTCCATGG
2416H4_v2       TTGCCAAGAGTC
2417G1_v2       GGTATGGCTACT
PCRzymo2        CAACACATGCTG
2417B9_v2       GCTTGAGCTTGA
2417D1_v2       AACATGCATGCC
2417C8_v2       TCGGCTTGGAAT
2417D2_v2       CTATGAGTCCAG
2238B3_v3       ATCTTGGAGTCG
2238B5_v3       GAAGGTGAAGGT
2238G11_v3      AGTCCACTGGTA
2238B9_v3       GCCGATTGTAAC
2238D2_v3       CGTAGCCAACAT
2238E12_v3      ACGCAATGTCTG
2238A3_v3       TAATCGGTGCCA
2416A9_v3       CGAGTCACGATT
PCRneg4 AGGTTCTTAGGC
2417G1_v3       CAGAATCGCTCA
2417C8_v3       TTCGCAGATACG
2417D2_v3       ACTCCGATAGAC
PCRzymo3        GATAGCACTCGT

barcodes.txt (showing whitespace in vim)

id^Ibarcode$
2238B3^IACCTGATCCGCA$
2238B5^IGTTGACCATCGC$
2238G11^ITGACTGCGTTAG$
2238B9^ICATACCGTGAGT$
2238D2^ICCGAGTACAATC$
PCRneg1^IAGCCTGGTACCT$
2238E12^IGAAGACAGCGAC$
2238F3^ITGGACTCAGCTA$
2238F6^IGTCCGATCCTAG$
2238A3^ITCATTAGCGTGG$
2238B6^ITAGGCTCGTGCT$
PCRzymo1^ICGGTAGTTGATC$
2416A9^IGTACTCGAACCA$
2416D2^IACTGCTATCGCG$
2416H4^ICACTGAGTACGT$
2417G1^ICCTAAGAGCATC$
2417B9^IATCGTGTGTTGG$
2417D1^ICGTCCGTATGAA$
2417C8^IGTGCAACCAATC$
2417D2^IACCAGTGACTCA$
PCRneg2^IGGATACTCGCAT$
blank1^ITAGTGCATTCGG$
blank2^IAGAACACGGAAG$
2238B3_v2^IGTCCGCAAGTTA$
2238B5_v2^ICCTGTCCTATCT$
2238G11_v2^ITTGTCAGCTGGA$
2238B9_v2^IGATAAGCGCCTT$
2238D2_v2^IACAGGTAGAGAG$
2238E12_v2^ICTCTATTCCACC$
2238F3_v2^IACCACGATGCTA$
PCRneg3^ITGGCGATACGTT$
2238F6_v2^IGCACTGGCATAT$
2238A3_v2^ITGTGCTTGTAGG$
2238B6_v2^ICACAACCACAAC$
2416A9_v2^IAGAGTCTTGCCA$
2416D2_v2^ICTGTGTCCATGG$
2416H4_v2^ITTGCCAAGAGTC$
2417G1_v2^IGGTATGGCTACT$
PCRzymo2^ICAACACATGCTG$
2417B9_v2^IGCTTGAGCTTGA$
2417D1_v2^IAACATGCATGCC$
2417C8_v2^ITCGGCTTGGAAT$
2417D2_v2^ICTATGAGTCCAG$
2238B3_v3^IATCTTGGAGTCG$
2238B5_v3^IGAAGGTGAAGGT$
2238G11_v3^IAGTCCACTGGTA$
2238B9_v3^IGCCGATTGTAAC$
2238D2_v3^ICGTAGCCAACAT$
2238E12_v3^IACGCAATGTCTG$
2238A3_v3^ITAATCGGTGCCA$
2416A9_v3^ICGAGTCACGATT$
PCRneg4^IAGGTTCTTAGGC$
2417G1_v3^ICAGAATCGCTCA$
2417C8_v3^ITTCGCAGATACG$
2417D2_v3^IACTCCGATAGAC$
PCRzymo3^IGATAGCACTCGT$

Qiime is installed via conda with the following info:

$ qiime info

System versions
Python version: 3.8.16
QIIME 2 release: 2023.2
QIIME 2 version: 2023.2.0
q2cli version: 2023.2.0

Installed plugins
alignment: 2023.2.0
composition: 2023.2.0
cutadapt: 2023.2.0
dada2: 2023.2.0
deblur: 2023.2.0
demux: 2023.2.0
diversity: 2023.2.0
diversity-lib: 2023.2.0
emperor: 2023.2.0
feature-classifier: 2023.2.0
feature-table: 2023.2.0
fragment-insertion: 2023.2.0
gneiss: 2023.2.0
longitudinal: 2023.2.0
metadata: 2023.2.0
phylogeny: 2023.2.0
quality-control: 2023.2.0
quality-filter: 2023.2.0
sample-classifier: 2023.2.0
taxa: 2023.2.0
types: 2023.2.0
vsearch: 2023.2.0

OS version:

$ sw_vers
ProductName:	macOS
ProductVersion:	12.6.3
BuildVersion:	21G419

[gregcaporaso]

Hi @wasteoftime,

@wasteoftime

It seems most likely that the cutadapt plugin isn't able to parse newlines on Macs when reading the barcodes even though somehow it IS able to determine when I have non-unique values in the ID column

Yes, it seems like QIIME 2 can handle the file fine. I suspect the issue is in cutadapt itself, not the QIIME 2 cutadapt plugin.

You can test that the file is ok for QIIME 2 by running:

qiime tools inspect-metadata barcodes.txt

If that command completes successfully, the file is formatted correctly for QIIME 2. However, because QIIME 2 sometimes calls other applications (cutacdapt, in this case) it's possible that what works for QIIME 2 won't always work for the underlying application.

This seems to be an issue with cutadapt's reading of your barcodes.txt file. I did some searching on the error message that cutadapt is generating and I came across this issue. You could try to address this as discussed in that issue.

Good luck!