q2-Sidle 2020.8 memory issue

Hi, jwdebelius

Yes, I'm doing a meta-analysis, I used 5 datasets and 1 of them needed the plugin q2-sidle due to it was sequenced using 16S 5R method.

I ran into another problem of q2-sidle this morning, which I noticed, has been discussed somewhere else in the Forum one year ago. But I thought I might still report my problem to see if there are available solutions now.

I ran the following command on a server with 512Gb RAM. After running the reconstruct 12 h, I checked the status with command top and found the usage of memory is 300Gb. And, after a total of 26 h, I got no output and an empty nohup.out, the process was killed, I guess it might still due to the memory issue.
I used q2-sidle-2020.8 and q2-2021.11.

nohup qiime sidle reconstruct-counts \
  --p-region P1 \
    --i-kmer-map sidle-db-P1-100nt-map.qza \
    --i-regional-alignment alignment/P1_align-map.qza \
    --i-regional-table P1_ISR_table-100nt.qza \
  --p-region P2 \
    --i-kmer-map sidle-db-P2-100nt-map.qza \
	--i-regional-alignment alignment/P2_align-map.qza \
    --i-regional-table P2_ISR_table-100nt.qza \
  --p-region P3 \
    --i-kmer-map sidle-db-P3-100nt-map.qza \
	--i-regional-alignment alignment/P3_align-map.qza \
    --i-regional-table P3_ISR_table-100nt.qza \
  --p-region P4 \
    --i-kmer-map sidle-db-P4-100nt-map.qza \
	--i-regional-alignment alignment/P4_align-map.qza \
    --i-regional-table P4_ISR_table-100nt.qza \
  --p-region P5 \
    --i-kmer-map sidle-db-P5-100nt-map.qza \
	--i-regional-alignment alignment/P5_align-map.qza \
    --i-regional-table P5_ISR_table-100nt.qza \
  --p-n-workers 16 \
  --o-reconstructed-table reconstruction/league_table.qza \
  --o-reconstruction-summary reconstruction/league_summary.qza \
  --o-reconstruction-map reconstruction/league_map.qza &

Is there any parameter I can define to avoid this problem? Or as I read the sidle documentary, I noticed there used to be a method reconstruct-database, will it be ok I switch to other version of sidle to finish reconstruct-counts with several different method?

Thanks,
Circle

-rw-rw-r--  141K    P1_align-map.qza
-rw-rw-r--  469K    P2_align-map.qza
-rw-rw-r--   73K    P3_align-map.qza
-rw-rw-r--  262K    P4_align-map.qza
-rw-rw-r--   65K    P5_align-map.qza
-rw-rw-r--  6.9M sidle-db-P1-100nt-map.qza
-rw-rw-r--  33M  sidle-db-P2-100nt-map.qza
-rw-rw-r--  55M  sidle-db-P3-100nt-map.qza
-rw-rw-r--  13M  sidle-db-P4-100nt-map.qza
-rw-rw-r--  18M  sidle-db-P5-100nt-map.qza
-rw-rw-r--  47K  P1_ISR_table-100nt.qza
-rw-rw-r--  62K  P2_ISR_table-100nt.qza
-rw-rw-r--  29K  P3_ISR_table-100nt.qza
-rw-rw-r--  63K  P4_ISR_table-100nt.qza
-rw-rw-r--  53K  P5_ISR_table-100nt.qza

Hi @Circle,

I moved this over ot a new topic, since its a separate issue.

You need to use the main branch, rather than the 2020.8 release. We haven't quite gotten around to cutting a new release (we will do one with the next qiime2 release), but the updated version has significant performance improvements.

Best,
Justine

Hi jwdebelius

Sorry for my late reply, other jobs popped up... I used the version of the main branch, the memory problem did not appear at all, thank you a lot for your reply.

Best,
Circle

1 Like