Unsigned integer overflow in `toom_eval_pm2.c`

[marco.bodrato at tutanota.com]

Ciao,

3 set 2023, 22:16 da nisse at lysator.liu.se:

> Andrew Teylu <andrewvaughanj at gmail.com> writes:
>


> I see no good reason to involve any signed values here, though. Maybe
> the variable neg, and the return value, should be changed to mp_limb_t,
> or at least unsigned int?
>

Yes, unsigned is the best choice, it used to store a positive or
negative number, but now it actually is a mask: 0 or ~0.

I attach a possible patch.


>> I guess maybe a different question is: what is that code supposed to
>> do? Is the intent to `xor` with `0xFFFFFFFF` if the `k` is even and
>> `xor` with `0` if the `k` is odd (i.e., it either flips all the bits
>> in the even case or leaves them in the odd case)?
>>
>
> I think intention is to conditionally flip all the bits. And in
> addition, neg should always be either all ones or all zeros.
>

Correct.

Ĝis,
m
-------------- next part --------------
A non-text attachment was scrubbed...
Name: unsigned.patch
Type: text/x-patch
Size: 7441 bytes
Desc: not available
URL: <https://gmplib.org/list-archives/gmp-bugs/attachments/20230904/a02fcbc9/attachment-0001.bin>