Merging Libraries-Same DADA2 Perim

Hi,

So I seem to be having the same issue I previously had (this post). However, I have processed all of the libraries in the same manner, using the same DADA2 perimeters. Originally, when I merged lib 1-4 there was no batch effect.
However, we had 10 samples from libraries 3 and 2 that previously failed to amplify and we manage to get these missing samples sequenced in what I am calling library 4E (extra). Anyway, when I merge these libraries I once again get a batch effect and where libraries 1, 2, 4, 4E are combined but 3 is separate. I have rerun the merging 2x to make sure I did it correctly and verified the metadata and I cannot seem to find the problem.

Do you have any ideas about what it would be or how I can go about fixing the issues?

I have attached the file below, if you color by the library you can see what I am referring to.

bray_curtis_emperor.qzv (1.0 MB)

Hi @fabipc,

That is weird to me! You definitely have a clear seperation with the library 3. But when I check your provenance, it looks like there’s one table where you didn’t do primer trimming (UUID 9bf41441-40b6-417f-afff-d9a5f03c878d). I don’t know if that’s your library 3, but that would be my guess. Maybe try trimming library 3 with cutadapt, denoising, then merging.

Best,
Justine

2 Likes

Oh so weird, I thought I did run trimming on it, but maybe I missed it. The weird thing is that prior to adding library 4E, library 1-4 merged perfectly well. I am going to go ahead and redo library 3 steps and merge the libraries and hope it works.

Thank you very much @Jwdebelius, I completely forgot to look at the provenance.

2 Likes

Hi,

So I redid library 3 and still had the same issue. I decided to look at the dada2 output for the 3rd library and by looking at the provenance file for library 3 (reprocessed) and comparing it to library 2, I can see that there is a step missing but I am not sure how to fix it, since I am literally using the exact same script to process all the libraries. Can you please help?

This is where I can see we are missing something but not sure how to fix it.

Library 3-that does not work
Screen Shot 2020-10-21 at 9.30.04 PM

Library 2-which worked
Screen Shot 2020-10-21 at 11.11.12 AM

Here is the dada2 output for library 2 and library 3 (the one with the issues). I would really appreciate any advice you have.

Fungal-2-Table-dada2-209-201.qzv (731.7 KB)
Fungal-3-Table-dada2-209-201.qzv (645.7 KB) .

Here is the script I used to process the 3rd library:

qiime dada2 denoise-paired
--i-demultiplexed-seqs Demux/Fungi-demux-paired-end-3.qza
--p-trunc-len-f 209
--p-trunc-len-r 201
--p-trim-left-f 1
--p-trim-left-r 1
--p-trunc-q 0
--p-n-threads 30
--o-table DADA2/Fungal-3-Table-dada2-209-201.qza
--o-representative-sequences DADA2/Fungal-3-RepSeqs-dada2-209-201.qza
--o-denoising-stats DADA2/Fungal-3-Stats-dada2-209-201.qza
--verbose

qiime metadata tabulate
--m-input-file DADA2/Fungal-3-Stats-dada2-209-201.qza
--o-visualization DADA2/Fungal-3-Stats-dada2-209-201.qzv

qiime feature-table summarize
--i-table DADA2/Fungal-3-Table-dada2-209-201.qza
--o-visualization DADA2/Fungal-3-Table-dada2-209-201.qzv
--m-sample-metadata-file Fungal-Metadata-Lib3.tsv

qiime feature-table tabulate-seqs
--i-data DADA2/Fungal-3-RepSeqs-dada2-209-201.qza
--o-visualization DADA2/Fungal-3-RepSeqs-dada2-209-201.qzv

Hello,

I am so sorry to post another question, but my last question seems to have been abandoned, and everything I have tried thus far failed to work.

I have 5 libraries that I need to merge (ITS2), and previously, I had merged 4 libraries and everything looked good. However when I add the 5th library, one of the libraries that previously worked appears to have batch effects. I had previously thought that there was an error with the DADA2 file for this library as it appeared to be missing a step, as per this question that I had previously made, and after rerunning DADA2 a few times, the error appeared to have fixed itself. However, after merging the files, when I look at the beta diversity, I can clearly see a batch effect and I am not sure how to fix it. It seems like it is still showing that one of the libraries did not get run through DADA2, as previously mentioned in the previous post I made. However, all libraries were run through DADA2, using the exact same code and perimeter.

I am really at a loss and I am unsure on how to process, can you please help.

#Library 3 is the one I am having issues with, and hence I added the table output for reference.

Fungal-Table-Merged-1-4E-MJ-NS.qzv (1.4 MB)
bray_curtis_emperor.qzv (1.0 MB)
Fungal-3-Table-dada2-209-201.qzv (685.2 KB)

Hi @fabipc,

I’m really sorry, I dropped the ball here. This year has not helped my juggling :woman_juggling: . If this happens again and its been a week and you haven’t heard back, please post again and tag the person again.

Based on your proveance, you have two libraris with different trimming parameters. I’m not great at detangling which artifact is which library ID (although it would be a super cool feature to figure out) so I’m not sure if it’s your problem child again or not.

In seems like, in general, this is a workflow management problem. I know there are some plans in QIIME 2 to clean that up, but it’s definately hard in the iterim! You might consider
parameterizing a jupyter notebook, shell script, or snakemake file as ways to make sure your parameters are consistent.

But, I think the trimming. parameters are the next place I’d check.

Best,
Justine

1 Like

hi @jwdebelius

As far as I know my parameters are all the same, but since I had no idea what else to do, I am currently re-running all the libraries and literally just copied and pasted the scripts over from one to another. I do use Jupyter so it works great. I am just waiting for the final output and crossing my fingers hoping it worked.

I will let you know.

1 Like

Hi @fabipc,

When I checked your provenance, there were two runs with different trimming parameters. So, I would check that. I’m not sure that you need to do a second trim 5’ trim in dada2 if you’ve already applied cutadapt, just as something to keep in mind?

I’m glad you’ve got a system that makes it easy! :smiley:

Best,
Justine

So, you were correct. After rerunning everything and re-creating the scripts to where I copied the scripts from one to another, instead of from the master scripts I was using (on word doc). I realized that in fact the library 3 had an error to where the trimming parameters were 1,1 instead of F=2, R=3. I can’t believe it took me so long to see this.

I am so sorry for all the post, I should have known better since I had this problem the first time I tried to merge, but I must of accidentally changed my master script at some point and did not realize it. Lesson learned!

3 Likes

Hi @fabipc,

I’m glad you solved it! And, I think sometimes tracking this stuff can be the hardest thing. I recently realized that I had a mistake in an analysis when i went back to write up the methods :scream:. Provenance definitely makes it a lot easier. I hope things are smoother going forward!

Best,
Justine

1 Like

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.