q2-Sidle 2020.8 memory issue

[Circle]

Hi， jwdebelius

Yes, I'm doing a meta-analysis, I used 5 datasets and 1 of them needed the plugin q2-sidle due to it was sequenced using 16S 5R method.

I ran into another problem of q2-sidle this morning, which I noticed, has been discussed somewhere else in the Forum one year ago. But I thought I might still report my problem to see if there are available solutions now.

I ran the following command on a server with 512Gb RAM. After running the reconstruct 12 h, I checked the status with command top and found the usage of memory is 300Gb. And, after a total of 26 h, I got no output and an empty nohup.out, the process was killed, I guess it might still due to the memory issue.
I used q2-sidle-2020.8 and q2-2021.11.

nohup qiime sidle reconstruct-counts \
  --p-region P1 \
    --i-kmer-map sidle-db-P1-100nt-map.qza \
    --i-regional-alignment alignment/P1_align-map.qza \
    --i-regional-table P1_ISR_table-100nt.qza \
  --p-region P2 \
    --i-kmer-map sidle-db-P2-100nt-map.qza \
	--i-regional-alignment alignment/P2_align-map.qza \
    --i-regional-table P2_ISR_table-100nt.qza \
  --p-region P3 \
    --i-kmer-map sidle-db-P3-100nt-map.qza \
	--i-regional-alignment alignment/P3_align-map.qza \
    --i-regional-table P3_ISR_table-100nt.qza \
  --p-region P4 \
    --i-kmer-map sidle-db-P4-100nt-map.qza \
	--i-regional-alignment alignment/P4_align-map.qza \
    --i-regional-table P4_ISR_table-100nt.qza \
  --p-region P5 \
    --i-kmer-map sidle-db-P5-100nt-map.qza \
	--i-regional-alignment alignment/P5_align-map.qza \
    --i-regional-table P5_ISR_table-100nt.qza \
  --p-n-workers 16 \
  --o-reconstructed-table reconstruction/league_table.qza \
  --o-reconstruction-summary reconstruction/league_summary.qza \
  --o-reconstruction-map reconstruction/league_map.qza &

Is there any parameter I can define to avoid this problem? Or as I read the sidle documentary, I noticed there used to be a method reconstruct-database, will it be ok I switch to other version of sidle to finish reconstruct-counts with several different method?

Thanks,
Circle

[Circle]

-rw-rw-r--  141K    P1_align-map.qza
-rw-rw-r--  469K    P2_align-map.qza
-rw-rw-r--   73K    P3_align-map.qza
-rw-rw-r--  262K    P4_align-map.qza
-rw-rw-r--   65K    P5_align-map.qza

-rw-rw-r--  6.9M sidle-db-P1-100nt-map.qza
-rw-rw-r--  33M  sidle-db-P2-100nt-map.qza
-rw-rw-r--  55M  sidle-db-P3-100nt-map.qza
-rw-rw-r--  13M  sidle-db-P4-100nt-map.qza
-rw-rw-r--  18M  sidle-db-P5-100nt-map.qza

-rw-rw-r--  47K  P1_ISR_table-100nt.qza
-rw-rw-r--  62K  P2_ISR_table-100nt.qza
-rw-rw-r--  29K  P3_ISR_table-100nt.qza
-rw-rw-r--  63K  P4_ISR_table-100nt.qza
-rw-rw-r--  53K  P5_ISR_table-100nt.qza

[jwdebelius (Justine Debelius)]

Hi @Circle,

I moved this over ot a new topic, since its a separate issue.

You need to use the main branch, rather than the 2020.8 release. We haven't quite gotten around to cutting a new release (we will do one with the next qiime2 release), but the updated version has significant performance improvements.

Best,
Justine

[Circle]

Hi jwdebelius

Sorry for my late reply, other jobs popped up... I used the version of the main branch, the memory problem did not appear at all, thank you a lot for your reply.

Best,
Circle