# Ion Torrent data denoise with dada2

**URL:** <https://forum.qiime2.org/t/ion-torrent-data-denoise-with-dada2/3224>\
**Category:** User Support\
**Tags:** dada2, ion-torrent\
**Created:** [February 27, 2018, 3:03pm UTC](https://forum.qiime2.org/t/ion-torrent-data-denoise-with-dada2/3224 "2018-02-27T15:03:21Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Marek\_Koutny](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/m/b2d939/32.png) [@Marek\_Koutny](https://forum.qiime2.org/u/Marek_Koutny)\
**Post date:** [February 27, 2018, 3:03pm UTC](https://forum.qiime2.org/t/ion-torrent-data-denoise-with-dada2/3224/1 "2018-02-27T15:03:21Z")

</div>

Hello qiime2 team,

I was happy to import my ion torrent data with help of this post:

> [@How to do multiplex of Ion torrent sequences](https://forum.qiime2.org/t/how-to-do-multiplex-of-ion-torrent-sequences/2907):
>
> Dear Qiime2 users, I am newbie to bioinformatic analysis and for qiime2 as well. We are using Ion torrent PGM platform for V4 16SrRNA region. After sequencing, output are all .fastq files (according to our samples) and one nomatch.fastq, kindly see the attachment ([nomatch\_fastq]. It seems our sequencer itself do demultiplex. My doubts are - As I understood in qiime2 tutorial, we need barcodes.fastq file but i could not able to find the same file in our output. I could not able to understand …

and obtained single-end-demux.qza artifact file.

But now I can not proceed through dada2 tool:  
qiime dada2 denoise-single \

> --i-demultiplexed-seqs single-end-demux.qza   
> --p-trim-left 15   
> --p-trunc-len 250   
> --o-representative-sequences rep-seqs.qza   
> --o-table table.qza

It does not like my data:  
Plugin error from dada2:

Argument to parameter 'demultiplexed\_seqs' is not a subtype of SampleData[PairedEndSequencesWithQuality | SequencesWithQuality].

Debug info has been saved to /tmp/qiime2-q2cli-err-8hez8tae.log

There is not much help in the log file. It says basically the same.

Thanks,  
Regards,  
Marek

---

<div class="post-metadata">

**Author:** ![colinbrislawn](https://forum.qiime2.org/user_avatar/forum.qiime2.org/colinbrislawn/32/6221_2.png) [@colinbrislawn](https://forum.qiime2.org/u/colinbrislawn)\
**Post date:** [February 27, 2018, 7:41pm UTC](https://forum.qiime2.org/t/ion-torrent-data-denoise-with-dada2/3224/3 "2018-02-27T19:41:13Z")

</div>

Hello Marek,

Thanks for posting all of this information. I think we are pretty close to figuring this out.

In Matt's post on that other thread, he suggests

```bash
qiime tools import \
 --type 'SampleData[JoinedSequencesWithQuality]' \
 --input-path se-33-manifest.csv \
 --output-path single-end-demux.qza \
 --source-format SingleEndFastqManifestPhred33

```

Did you use this command, or have you tried using another `--type`?

Colin

---

<div class="post-metadata">

**Author:** ![thermokarst](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@thermokarst](https://forum.qiime2.org/u/thermokarst)\
**Post date:** [February 27, 2018, 7:50pm UTC](https://forum.qiime2.org/t/ion-torrent-data-denoise-with-dada2/3224/4 "2018-02-27T19:50:47Z")

</div>

Hi @Marek_Koutny, to elaborate a bit on @colinbrislawn's suggestion, what format are your reads in, and how did you import them? Please copy and paste your import command here. Also, what are the results when you run:

```bash
qiime tools peek single-end-demux.qza

```

That error message you posted means that the file `single-end-demux.qza` doesn't contain `SampleData[PairedEndSequencesWithQuality]` or `SampleData[SequencesWithQuality]`, which leaves @colinbrislawn's suggestion that these data are `SampleData[JoinedSequencesWithQuality]`. If that is the case (and the `peek` command above will confirm that for us), that means you shouldn't use DADA2, because that technique assumes that your paired-end reads aren't pre-joined. You can look at using q2-deblur, check [out this tutorial](https://docs.qiime2.org/2018.2/tutorials/read-joining/) for more info. Thanks! 🦖

---

<div class="post-metadata">

**Author:** ![Marek\_Koutny](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/m/b2d939/32.png) [@Marek\_Koutny](https://forum.qiime2.org/u/Marek_Koutny)\
**Post date:** [February 27, 2018, 7:51pm UTC](https://forum.qiime2.org/t/ion-torrent-data-denoise-with-dada2/3224/6 "2018-02-27T19:51:16Z")

</div>

Hi Colin,

Thanks for answering. I have used exactly that one you quote.

Marek

---

<div class="post-metadata">

**Author:** ![Marek\_Koutny](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/m/b2d939/32.png) [@Marek\_Koutny](https://forum.qiime2.org/u/Marek_Koutny)\
**Post date:** [February 28, 2018, 12:27pm UTC](https://forum.qiime2.org/t/ion-torrent-data-denoise-with-dada2/3224/7 "2018-02-28T12:27:26Z")

</div>

Hi Mattew,

Thanks for helping.

The result of peek is:  
qiime tools peek single-end-demux.qza  
UUID: 88af3b88-0836-4069-96bc-8dc7a51f28c3  
Type: SampleData[JoinedSequencesWithQuality]  
Data format: SingleLanePerSampleSingleEndFastqDirFmt

and I have used this for import:  
qiime tools import   
--type 'SampleData[JoinedSequencesWithQuality]'   
--input-path manifest.csv   
--output-path single-end-demux.qza   
--source-format SingleEndFastqManifestPhred33

I assume I have pre-joined data  
I have constructed manifest as recommended with forward at the end for all fastq files.

Regards,

Marek

---

<div class="post-metadata">

**Author:** ![thermokarst](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@thermokarst](https://forum.qiime2.org/u/thermokarst)\
**Post date:** [February 28, 2018, 12:30pm UTC](https://forum.qiime2.org/t/ion-torrent-data-denoise-with-dada2/3224/8 "2018-02-28T12:30:10Z")

</div>

Hi @Marek_Koutny --- cool, this all makes sense, it looks like you had pre-joined reads, and you imported them as `SampleData[JoinedSequencesWithQuality]`. So as I mentioned before, you won't be able to work with DADA2, because it operates on paired-end reads before they are joined, but you have a few other analysis options available to you. Please check out the tutorial [I linked to above](https://forum.qiime2.org/t/ion-torrent-data-denoise-with-dada2/3224/4). Keep us posted! 🦖

---

<div class="post-metadata">

**Author:** ![Marek\_Koutny](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/m/b2d939/32.png) [@Marek\_Koutny](https://forum.qiime2.org/u/Marek_Koutny)\
**Post date:** [March 6, 2018, 3:04pm UTC](https://forum.qiime2.org/t/ion-torrent-data-denoise-with-dada2/3224/9 "2018-03-06T15:04:14Z")

</div>

Hi Matthew. With you help I have get little bit further. I was able to get my data through deblur and get feature-table and rep-seq file. I have still several issues. In the feature table I can see that my feature sequences are in one sample only, which is weird, these are samples are from the same soil in different times. Please, can I create some table with feature sequences, frequencies and sample numbers? I cannot see which feature was found in which sample. Thank you, Marek.

---

<div class="post-metadata">

**Author:** ![ebolyen](https://forum.qiime2.org/user_avatar/forum.qiime2.org/ebolyen/32/11_2.png) [@ebolyen](https://forum.qiime2.org/u/ebolyen)\
**Post date:** [March 6, 2018, 3:35pm UTC](https://forum.qiime2.org/t/ion-torrent-data-denoise-with-dada2/3224/10 "2018-03-06T15:35:12Z")

</div>



---

<div class="post-metadata">

**Author:** ![thermokarst](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@thermokarst](https://forum.qiime2.org/u/thermokarst)\
**Post date:** [March 6, 2018, 10:11pm UTC](https://forum.qiime2.org/t/ion-torrent-data-denoise-with-dada2/3224/11 "2018-03-06T22:11:38Z")

</div>

Hi @Marek_Koutny, can you please provide the visualization from [`demux summarize`](https://docs.qiime2.org/2018.2/plugins/available/demux/summarize/), as well as the commands you ran for processing through deblur? Please include any intermediate commands, too. Thanks! 🦖

---

<div class="post-metadata">

**Author:** ![thermokarst](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@thermokarst](https://forum.qiime2.org/u/thermokarst)\
**Post date:** [March 6, 2018, 10:11pm UTC](https://forum.qiime2.org/t/ion-torrent-data-denoise-with-dada2/3224/12 "2018-03-06T22:11:39Z")

</div>



---

<div class="post-metadata">

**Author:** ![MMC\_northS](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/m/a9adbd/32.png) [@MMC\_northS](https://forum.qiime2.org/u/MMC_northS)\
**Post date:** [March 13, 2018, 9:58am UTC](https://forum.qiime2.org/t/ion-torrent-data-denoise-with-dada2/3224/13 "2018-03-13T09:58:13Z")

</div>

Hello @thermokarst

I am trying similar things that @Marek_Koutny but I imported my data using --type 'SampleData[SequencesWithQuality]' and manifest\_file.

i used that one because I do not have paired-end sequences so I assume that they do not pre-joined (nothing to join). In this case I can use dada2 for single-end option? or it is most adequate use deblur denoise?

Other question that I have is if it is good use deblur or I can use vsearch with open-reference-otu-picking directly after filtering by quality. Because I do not want trim my sequences and I can filter by number of reads or abundance later.

Thank you very much for your help in advance!  
MMC

---

<div class="post-metadata">

**Author:** ![Nicholas\_Bokulich](https://forum.qiime2.org/user_avatar/forum.qiime2.org/nicholas_bokulich/32/19937_2.png) [@Nicholas\_Bokulich](https://forum.qiime2.org/u/Nicholas_Bokulich)\
**Post date:** [March 13, 2018, 1:08pm UTC](https://forum.qiime2.org/t/ion-torrent-data-denoise-with-dada2/3224/14 "2018-03-13T13:08:21Z")

</div>



---

<div class="post-metadata">

**Author:** ![thermokarst](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@thermokarst](https://forum.qiime2.org/u/thermokarst)\
**Post date:** [March 13, 2018, 9:43pm UTC](https://forum.qiime2.org/t/ion-torrent-data-denoise-with-dada2/3224/15 "2018-03-13T21:43:26Z")

</div>

Hi @MMC_northS!

> [@MMC\_northS](#):
>
> i used that one because I do not have paired-end sequences so I assume that they do not pre-joined (nothing to join). In this case I can use dada2 for single-end option? or it is most adequate use deblur denoise?

If they are single end, but pre-joined from the ion torrent processing step, you will need to import them as `SampleData[JoinedSequencesWithQuality]`, otherwise you can import them as `SampleData[SequencesWithQuality]` --- we have no way of know about what _kind_ of data your reads represent, so this is something you will need to take care of understanding prior to importing.

> [@MMC\_northS](#):
>
> Other question that I have is if it is good use deblur or I can use vsearch with open-reference-otu-picking directly after filtering by quality. Because I do not want trim my sequences and I can filter by number of reads or abundance later.

You are good to go either direction. Just to clarify though, there is no requirement that you trim your reads, per se - its just that deblur expects your reads to be a consistent length. Depending on the sequencing technology used, you can get this without any trimming.

Hope that helps! 🦖

---

<div class="post-metadata">

**Author:** ![thermokarst](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@thermokarst](https://forum.qiime2.org/u/thermokarst)\
**Post date:** [March 13, 2018, 9:43pm UTC](https://forum.qiime2.org/t/ion-torrent-data-denoise-with-dada2/3224/16 "2018-03-13T21:43:27Z")

</div>



---

<div class="post-metadata">

**Author:** ![MMC\_northS](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/m/a9adbd/32.png) [@MMC\_northS](https://forum.qiime2.org/u/MMC_northS)\
**Post date:** [March 14, 2018, 9:36am UTC](https://forum.qiime2.org/t/ion-torrent-data-denoise-with-dada2/3224/17 "2018-03-14T09:36:37Z")

</div>

Hi @thermokarst

Thanks for your answers!

I understand that my data are not pre-joined. I used one single-end kit for PGM and the sequencer, after quality sequencer filters, made the demultiplexing cutting adapters and barcodes and giving us one fastq file per sample. I understand from that processing that my sequences were not pre-joined, so I used SampleData[SequencesWithQuality]. Is ths right?

Regarding the deblur (or dada) analyses, I do not understand what do you mean with "consistent length"? I understand you are speaking about the trimming parameters in those analyses but I would prefer do not cut my sequences. In that case could I use the dereplication step (Vsearch plugin) and then directly open-reference otu picking?

Thank you so much for your clarifications!  
MMC

---

<div class="post-metadata">

**Author:** ![ebolyen](https://forum.qiime2.org/user_avatar/forum.qiime2.org/ebolyen/32/11_2.png) [@ebolyen](https://forum.qiime2.org/u/ebolyen)\
**Post date:** [March 14, 2018, 3:08pm UTC](https://forum.qiime2.org/t/ion-torrent-data-denoise-with-dada2/3224/18 "2018-03-14T15:08:35Z")

</div>



---

<div class="post-metadata">

**Author:** ![thermokarst](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@thermokarst](https://forum.qiime2.org/u/thermokarst)\
**Post date:** [March 15, 2018, 1:03am UTC](https://forum.qiime2.org/t/ion-torrent-data-denoise-with-dada2/3224/19 "2018-03-15T01:03:22Z")

</div>

> [@MMC\_northS](#):
>
> I understand that my data are not pre-joined. I used one single-end kit for PGM and the sequencer, after quality sequencer filters, made the demultiplexing cutting adapters and barcodes and giving us one fastq file per sample. I understand from that processing that my sequences were not pre-joined, so I used SampleData[SequencesWithQuality]. Is ths right?

Yes (please see [my answer above](https://forum.qiime2.org/t/ion-torrent-data-denoise-with-dada2/3224/15) for more details).

> [@MMC\_northS](#):
>
> Regarding the deblur (or dada) analyses, I do not understand what do you mean with “consistent length”? I understand you are speaking about the trimming parameters in those analyses but I would prefer do not cut my sequences.

Some sequencing technologies/amplicons/primers/etc produce consistent length sequences (as in, all sequences are the same length, while some technologies/amplicons/primers/etc don't produce consistent length sequences. As I mentioned above - there is no strict requirement for DADA2 or deblur to trim your sequences ("there is no requirement that you trim your reads"), but often times people need to in order to process using those tools (because the nature of sequences are so variable, as I just mentioned). So, you don't have to trim, but those parameters are there for people that need it. I _highly_ recommend you spend some more time reading about DADA2, deblur, and the QIIME 2 docs --- all of your questions have answers in those documents.

> [@MMC\_northS](#):
>
> In that case could I use the dereplication step (Vsearch plugin) and then directly open-reference otu picking?

This is unrelated to trimming - you can use `vsearch cluster-features-open-reference` regardless of if you want to trim your sequences or not. I answered this above when I said "You are good to go either direction," my apologies if that wasn't clear.

---

<div class="post-metadata">

**Author:** ![thermokarst](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@thermokarst](https://forum.qiime2.org/u/thermokarst)\
**Post date:** [March 15, 2018, 1:03am UTC](https://forum.qiime2.org/t/ion-torrent-data-denoise-with-dada2/3224/20 "2018-03-15T01:03:30Z")

</div>



---

<div class="post-metadata">

**Author:** ![Marek\_Koutny](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/m/b2d939/32.png) [@Marek\_Koutny](https://forum.qiime2.org/u/Marek_Koutny)\
**Post date:** [March 15, 2018, 7:19am UTC](https://forum.qiime2.org/t/ion-torrent-data-denoise-with-dada2/3224/21 "2018-03-15T07:19:41Z")

</div>

Hello @THERMOKARST and others. Sorry for not responding for some time. I have get somewhat further. Now I have my taxonomy assigned and I can explore my result. Great! The main issue I had to solve was that my data still contained barcodes (i was told that they were trimmed) I have found this by visualizing the sequences with jalview, it is a great tool.  
I had to use greengene classifier, the both Silva classifiers provided produced errors, actually also Atacama data provided gave the same error with Silva.  
Details:  
(qiime2-2018.2) qiime2@qiime2core2018-2:~/AtacamaSoil$ qiime feature-classifier classify-sklearn \

> --i-classifier silva-119-99-nb-classifier.qza   
> --i-reads rep-seqs.qza   
> --o-classification taxonomy.qza  
> Plugin error from feature-classifier:

Debug info has been saved to /tmp/qiime2-q2cli-err-va152nns.log  
File content:  
Traceback (most recent call last):  
File "/home/qiime2/miniconda/envs/qiime2-2018.2/lib/python3.5/site-packages/q2cli/commands.py", line 246, in **call**  
results = action(\*\*arguments)  
File "", line 2, in classify\_sklearn  
File "/home/qiime2/miniconda/envs/qiime2-2018.2/lib/python3.5/site-packages/qiime2/sdk/action.py", line 222, in bound\_callable  
spec.view\_type, recorder)  
File "/home/qiime2/miniconda/envs/qiime2-2018.2/lib/python3.5/site-packages/qiime2/sdk/result.py", line 261, in \_view  
result = transformation(self.\_archiver.data\_dir)  
File "/home/qiime2/miniconda/envs/qiime2-2018.2/lib/python3.5/site-packages/qiime2/core/transform.py", line 59, in transformation  
new\_view = transformer(view)  
File "/home/qiime2/miniconda/envs/qiime2-2018.2/lib/python3.5/site-packages/q2\_feature\_classifier/\_taxonomic\_classifier.py", line 72, in \_1  
pipeline = joblib.load(os.path.join(dirname, 'sklearn\_pipeline.pkl'))  
File "/home/qiime2/miniconda/envs/qiime2-2018.2/lib/python3.5/site-packages/sklearn/externals/joblib/numpy\_pickle.py", line 578, in load  
obj = \_unpickle(fobj, filename, mmap\_mode)  
File "/home/qiime2/miniconda/envs/qiime2-2018.2/lib/python3.5/site-packages/sklearn/externals/joblib/numpy\_pickle.py", line 508, in \_unpickle  
obj = unpickler.load()  
File "/home/qiime2/miniconda/envs/qiime2-2018.2/lib/python3.5/pickle.py", line 1043, in load  
dispatchkey[0]  
File "/home/qiime2/miniconda/envs/qiime2-2018.2/lib/python3.5/site-packages/sklearn/externals/joblib/numpy\_pickle.py", line 341, in load\_build  
self.stack.append(array\_wrapper.read(self))  
File "/home/qiime2/miniconda/envs/qiime2-2018.2/lib/python3.5/site-packages/sklearn/externals/joblib/numpy\_pickle.py", line 184, in read  
array = self.read\_array(unpickler)  
File "/home/qiime2/miniconda/envs/qiime2-2018.2/lib/python3.5/site-packages/sklearn/externals/joblib/numpy\_pickle.py", line 130, in read\_array  
array = unpickler.np.empty(count, dtype=self.dtype)  
MemoryError

Up to now exploring my results I am surprised and a bit disappointed that my results do not match my previous analysis. As my samples comes from the same locations I would expect that the general pattern of microorganisms will be identical. However it seems that there is no such pattern. I thing it must be some artifact of the analysis or data processing. Please do you have some explanation?

---

<div class="post-metadata">

**Author:** ![thermokarst](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@thermokarst](https://forum.qiime2.org/u/thermokarst)\
**Post date:** [March 15, 2018, 12:55pm UTC](https://forum.qiime2.org/t/ion-torrent-data-denoise-with-dada2/3224/22 "2018-03-15T12:55:41Z")

</div>



[Next page](https://forum.qiime2.org/t/ion-torrent-data-denoise-with-dada2/3224.md?page=2)
