Unsigned integer overflow in `toom_eval_pm2.c`

[marco.bodrato at tutanota.com]

Ciao,

4 set 2023, 08:02 da nisse at lysator.liu.se:

> As Vincent suggested, it would be good to document somewhere what the
> convention is (values 0 or ~0).
>

Would a comment "It returns 0 or ~0, depending on the sign of the result"
added to all the mpn_toom_eval_ functions suffice?


> And maybe change to 0 and 1 convention,
>

I see a line

mpn/generic/toom42_mul.c:  vm1_neg = mpn_toom_eval_dgr3_pm1 (as1, asm1, ap, n, s, a0_a2) & 1;

where we actually use 0 or 1, and many lines like

mpn/generic/toom52_mul.c:  flags = (enum toom6_flags) (toom6_vm2_neg & mpn_toom_eval_pm2 (as2, asm2, 4, ap, n, s, a1a3));

where we use the value as a mask: 0 or ~ 0.


>From the value in {0,1} we can get mask = -value, and from the mask in {0, ~0} we can get value = mask & 1.
Of course we can switch to a different convention, if someone feels it's important. I don't.


> since that fits better with assigning it from the return value from
> mpn_sub_n, and it seems generally more consistent with how we handle
>

We never use mpn_sub_n to assign that value.


> boolean values elsewhere. But may need further changes, like to
> abs_sub_add_n, also noted by Vincent.
>
Yes, Vincent is right. New patch.

Should the value ~0 be written as ~0U to be correctly assigned to an unsigned?

Ĝis,
m
-------------- next part --------------
A non-text attachment was scrubbed...
Name: unsigned.patch
Type: text/x-patch
Size: 8650 bytes
Desc: not available
URL: <https://gmplib.org/list-archives/gmp-bugs/attachments/20230904/b8c33050/attachment.bin>