Hello @cduvallet ,
I am trying to explore q2-perc-norm to handle batch effects for my analysis.
I am using q2-2022.11 and installed the q2-perc-norm plugin using the provided github repo.
my metadata looks like below. It is merged from different batches/runs. id batch condition
batch1 case
batch1 case
batch1 control
batch1 control
batch2 case
batch2 case
batch2 control
batch2 control
... ...
I have about 11 batches (with same case/control setup per batch). I processed each batch (with similar parameters using deblur). I merged the individual tables after assigning taxonomy and then tried to apply percentile normalization.
At first, the error was related to the qiime feature table type (error message below)
There was a problem with the command:
(1/1) Invalid value for '--i-table': Expected an artifact of at least type
FeatureTable[RelativeFrequency]. An artifact of type FeatureTable[Frequency]
was provided.
I normalized the table by using relative frequency and tried applying perc-norm but again there was another error
Plugin error from perc-norm:
'DataFrame' object has no attribute 'to_dense'
Debug info has been saved to /var/folders/lf/8_nlx7g91m98c96ngq805_ch0000gn/T/qiime2-q2cli-err-xhj_ok3v.log
I can upload the debug info if you will need it to look into the issue.
I am not sure what I am doing wrong.
I repeated the above process on a single batch but got the same 2 errors.
Can you advice on what I should do to get this running?
This is a known issue related to pandas updates since the original plugin was written.
If you're open to being my guinea pig to test my changes, I'm happy to fix it in the source code so others will benefit as well. If you'd rather just fix it yourself, a local fix is described in the issue. Alternatively, if you're interested in learning how to collaborate on open-source code, I'm also happy to walk you through making & contributing the fix to this plugin!
Hello Claire,
Thank you for reaching out this quick.
I have seen the local fix but I will let it pass
It will be a thrilling experience to be the guinea pig but I will choose, first, to learn to collaborate on open-source code. If that takes unexpected turns, then we can explore the guinea pig route
Please let me know what our next steps are.
Hello @gregcaporaso
I am trying to use q2-perc-norm but I am getting an error message like below:
Traceback (most recent call last):
File "/opt/anaconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/q2cli/commands.py", line 352, in call
results = action(**arguments)
File "", line 2, in percentile_normalize
File "/opt/anaconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/qiime2/sdk/action.py", line 234, in bound_callable
outputs = self.callable_executor(scope, callable_args,
File "/opt/anaconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/qiime2/sdk/action.py", line 381, in callable_executor
output_views = self._callable(**view_args)
File "/opt/anaconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/perc_norm-2019.4.1-py3.8.egg/q2_perc_norm/_percentile_normalize.py", line 189, in percentile_normalize
norm_df = _percentile_normalize_one_df(
File "/opt/anaconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/perc_norm-2019.4.1-py3.8.egg/q2_perc_norm/_percentile_normalize.py", line 52, in _percentile_normalize_one_df
data=np.random.uniform(0.0, zero_val, size=(df.shape[0], df.shape[1])),
File "mtrand.pyx", line 1119, in numpy.random.mtrand.RandomState.uniform
OverflowError: Range exceeds valid bounds
I found this similar issue that was not resolved. Did you make any progress tracking the developer and coming up with a solution? Is there any way I could help to resolve?
Update to the to_dense error for those following along:
we have made the necessary changes to q2-perc-norm, so that it works with pandas >= 1.0 now
I was not able to publish the updated package to conda (and as I am no longer active in the field, don't have time to troubleshoot the conda build), so the installation instructions are now solely via local install rather than via conda
