Demultiplexing in AWS - Killed

[jnie93 (James Nie)]

Hi there!

I'm trying to run the “Atacama soil microbiome” tutorial on a m1x.large with paired end data that is multiplexed. I've been having an issue where when I run the demultiplexing step, I get a readout in Terminal that just says "Killed."

This is my exact sequence for how I run Qiime2 2018.6 on AWS. I try to follow the directions in the documentation as much as possible:

1. If I run exactly according to instructions, I get a "No Space Error." I've gotten around that by using the following function: " sudo umount /tmp" followed by " sudo mount -t tmpfs -o size=120G,mode=1777 overflow /tmp." Then I am able to run the tutorial up to the demultiplexing step.
2. When I run the demultiplexing step, it will run for a few hours, then it will say "Killed," with no error message written to the temporary folder.
3. I thought this may have been a memory issue, so I've tried increasing the tmpfs size from the step above up to 200 GB, but that didn't make a difference. I've also tried running more powerful AWS instances (m3x.large, etc), but that also didn't make a difference. I'm also only using 30% of the instance's CPU at max before it stops working. Any suggestions?

[thermokarst]

Hey there @jnie93!

@jnie93

If I run exactly according to instructions, I get a “No Space Error.” I’ve gotten around that by using the following function: " sudo umount /tmp " followed by " sudo mount -t tmpfs -o size=120G,mode=1777 overflow /tmp ." Then I am able to run the tutorial up to the demultiplexing step.

makes sense, the defaults probably aren't sufficient, but since they will differ from study to study, we leave config like this up to you.

@jnie93

When I run the demultiplexing step, it will run for a few hours, then it will say “Killed,” with no error message written to the temporary folder.

This sounds a lot like an out of memory error.

@jnie93

I thought this may have been a memory issue, so I’ve tried increasing the tmpfs size from the step above up to 200 GB

I think you might be confused, memory typically refers to RAM, not disk size. In this case, you are using a tmpfs facility, which conflates the two --- you are dedicating memory to be uses as temp disk space. So it looks like by increasing that number, you are actually making things worse by dedicating more of your RAM to disk space, when in actuality you probably need to be using less. One option is to stop using the tmpfs facility and instead just mount some portion of your hard drive as temp, allowing you to use all of your RAM as RAM.

[jnie93 (James Nie)]

Thank you so much! The demultiplexing step just worked!!

On to the next issue haha. In the next step, I’m running into issue when I try to visualize the demux.qza file as demux.qzv. When I run:

qiime demux summarize
–i-data demux.qza
–o-visualization demux.qzv

I get an error, with the readout as:
Traceback (most recent call last):
File “/home/qiime2/miniconda/envs/qiime2-2018.8/bin/qiime”, line 11, in
sys.exit(qiime())
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/click/core.py”, line 722, in call
return self.main(*args, **kwargs)
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/click/core.py”, line 697, in main
rv = self.invoke(ctx)
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/click/core.py”, line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/click/core.py”, line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/click/core.py”, line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/click/core.py”, line 535, in invoke
return callback(*args, **kwargs)
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/q2cli/commands.py”, line 244, in call
arguments, missing_in, verbose, quiet = self.handle_in_params(kwargs)
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/q2cli/commands.py”, line 326, in handle_in_params
kwargs, fallback=cmd_fallback)
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/q2cli/handlers.py”, line 375, in get_value
artifact = qiime2.sdk.Result.load(path)
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/qiime2/sdk/result.py”, line 65, in load
archiver = archive.Archiver.load(filepath)
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/qiime2/core/archive/archiver.py”, line 293, in load
archive = cls.get_archive(filepath)
File “/home/qiime2/miniconda/envs/qiime2-2018.8/lib/python3.5/site-packages/qiime2/core/archive/archiver.py”, line 258, in get_archive
raise ValueError("%s is not a QIIME archive." % filepath)

Any idea what’s happening here? I’m just copying and pasting the command from the tutorial - not sure what could be going wrong. I checked my /tmp folder and I’m seeing qiime2-archive-[gibberish] directories mentioned in this post (Not a qiime archive). One appeared within 5 minutes of starting demux and another appeared within 10 minutes of the creation of the demux.qza file.

I’m re-running the demux step now in case it was because the SSH connection was terminated, but within the first 5 minutes, a qiime2-archive-[gibberish] appeared in my /tmp folder.

I’m wondering if this is just a glitch or if this is an error I’m making on my end. Thanks!

[thermokarst]

@jnie93

I’m wondering if this is just a glitch or if this is an error I’m making on my end. Thanks!

For some reason QIIME 2 isn't able to save the demux.qzv file, because it is corrupt. Perhaps related to this:

@jnie93

in case it was because the SSH connection was terminated

?

demux summarize should run pretty quickly (minutes), you could also run this locally, too.

Try again and let us know how it goes. If it is still failing, send me a link to demux.qza in a DM and I can take a look!

:qiime2: