Skip to content

Commit 882d809

Browse files
vstinnervstinner
authored
bpo-46906: Add PyFloat_Pack8() to the C API (GH-31657)
Add new functions to pack and unpack C double (serialize and deserialize): * PyFloat_Pack2(), PyFloat_Pack4(), PyFloat_Pack8() * PyFloat_Unpack2(), PyFloat_Unpack4(), PyFloat_Unpack8() Document these functions and add unit tests. Rename private functions and move them from the internal C API to the public C API: * _PyFloat_Pack2() => PyFloat_Pack2() * _PyFloat_Pack4() => PyFloat_Pack4() * _PyFloat_Pack8() => PyFloat_Pack8() * _PyFloat_Unpack2() => PyFloat_Unpack2() * _PyFloat_Unpack4() => PyFloat_Unpack4() * _PyFloat_Unpack8() => PyFloat_Unpack8() Replace the "unsigned char*" type with "char*" which is more common and easy to use.
1 parent ecfff63 commit 882d809

File tree

13 files changed

+294
-91
lines changed

13 files changed

+294
-91
lines changed

Diff for: ‎Doc/c-api/float.rst

+82
Original file line numberDiff line numberDiff line change
@@ -76,3 +76,85 @@ Floating Point Objects
7676
.. c:function:: double PyFloat_GetMin()
7777
7878
Return the minimum normalized positive float *DBL_MIN* as C :c:type:`double`.
79+
80+
81+
Pack and Unpack functions
82+
=========================
83+
84+
The pack and unpack functions provide an efficient platform-independent way to
85+
store floating-point values as byte strings. The Pack routines produce a bytes
86+
string from a C :c:type:`double`, and the Unpack routines produce a C
87+
:c:type:`double` from such a bytes string. The suffix (2, 4 or 8) specifies the
88+
number of bytes in the bytes string.
89+
90+
On platforms that appear to use IEEE 754 formats these functions work by
91+
copying bits. On other platforms, the 2-byte format is identical to the IEEE
92+
754 binary16 half-precision format, the 4-byte format (32-bit) is identical to
93+
the IEEE 754 binary32 single precision format, and the 8-byte format to the
94+
IEEE 754 binary64 double precision format, although the packing of INFs and
95+
NaNs (if such things exist on the platform) isn't handled correctly, and
96+
attempting to unpack a bytes string containing an IEEE INF or NaN will raise an
97+
exception.
98+
99+
On non-IEEE platforms with more precision, or larger dynamic range, than IEEE
100+
754 supports, not all values can be packed; on non-IEEE platforms with less
101+
precision, or smaller dynamic range, not all values can be unpacked. What
102+
happens in such cases is partly accidental (alas).
103+
104+
.. versionadded:: 3.11
105+
106+
Pack functions
107+
--------------
108+
109+
The pack routines write 2, 4 or 8 bytes, starting at *p*. *le* is an
110+
:c:type:`int` argument, non-zero if you want the bytes string in little-endian
111+
format (exponent last, at ``p+1``, ``p+3``, or ``p+6`` ``p+7``), zero if you
112+
want big-endian format (exponent first, at *p*).
113+
114+
Return value: ``0`` if all is OK, ``-1`` if error (and an exception is set,
115+
most likely :exc:`OverflowError`).
116+
117+
There are two problems on non-IEEE platforms:
118+
119+
* What this does is undefined if *x* is a NaN or infinity.
120+
* ``-0.0`` and ``+0.0`` produce the same bytes string.
121+
122+
.. c:function:: int PyFloat_Pack2(double x, unsigned char *p, int le)
123+
124+
Pack a C double as the IEEE 754 binary16 half-precision format.
125+
126+
.. c:function:: int PyFloat_Pack4(double x, unsigned char *p, int le)
127+
128+
Pack a C double as the IEEE 754 binary32 single precision format.
129+
130+
.. c:function:: int PyFloat_Pack8(double x, unsigned char *p, int le)
131+
132+
Pack a C double as the IEEE 754 binary64 double precision format.
133+
134+
135+
Unpack functions
136+
----------------
137+
138+
The unpack routines read 2, 4 or 8 bytes, starting at *p*. *le* is an
139+
:c:type:`int` argument, non-zero if the bytes string is in little-endian format
140+
(exponent last, at ``p+1``, ``p+3`` or ``p+6`` and ``p+7``), zero if big-endian
141+
(exponent first, at *p*).
142+
143+
Return value: The unpacked double. On error, this is ``-1.0`` and
144+
:c:func:`PyErr_Occurred` is true (and an exception is set, most likely
145+
:exc:`OverflowError`).
146+
147+
Note that on a non-IEEE platform this will refuse to unpack a bytes string that
148+
represents a NaN or infinity.
149+
150+
.. c:function:: double PyFloat_Unpack2(const unsigned char *p, int le)
151+
152+
Unpack the IEEE 754 binary16 half-precision format as a C double.
153+
154+
.. c:function:: double PyFloat_Unpack4(const unsigned char *p, int le)
155+
156+
Unpack the IEEE 754 binary32 single precision format as a C double.
157+
158+
.. c:function:: double PyFloat_Unpack8(const unsigned char *p, int le)
159+
160+
Unpack the IEEE 754 binary64 double precision format as a C double.

Diff for: ‎Doc/whatsnew/3.11.rst

+6
Original file line numberDiff line numberDiff line change
@@ -766,6 +766,12 @@ New Features
766766
available directly (via :c:type:`PyCMethod`).
767767
(Contributed by Petr Viktorin in :issue:`46613`.)
768768

769+
* Add new functions to pack and unpack C double (serialize and deserialize):
770+
:c:func:`PyFloat_Pack2`, :c:func:`PyFloat_Pack4`, :c:func:`PyFloat_Pack8`,
771+
:c:func:`PyFloat_Unpack2`, :c:func:`PyFloat_Unpack4` and
772+
:c:func:`PyFloat_Unpack8`.
773+
(Contributed by Victor Stinner in :issue:`46906`.)
774+
769775

770776
Porting to Python 3.11
771777
----------------------

Diff for: ‎Include/cpython/floatobject.h

+9
Original file line numberDiff line numberDiff line change
@@ -10,3 +10,12 @@ typedef struct {
1010
// Macro version of PyFloat_AsDouble() trading safety for speed.
1111
// It doesn't check if op is a double object.
1212
#define PyFloat_AS_DOUBLE(op) (((PyFloatObject *)(op))->ob_fval)
13+
14+
15+
PyAPI_FUNC(int) PyFloat_Pack2(double x, char *p, int le);
16+
PyAPI_FUNC(int) PyFloat_Pack4(double x, char *p, int le);
17+
PyAPI_FUNC(int) PyFloat_Pack8(double x, char *p, int le);
18+
19+
PyAPI_FUNC(double) PyFloat_Unpack2(const char *p, int le);
20+
PyAPI_FUNC(double) PyFloat_Unpack4(const char *p, int le);
21+
PyAPI_FUNC(double) PyFloat_Unpack8(const char *p, int le);

Diff for: ‎Include/internal/pycore_floatobject.h

-48
Original file line numberDiff line numberDiff line change
@@ -38,54 +38,6 @@ struct _Py_float_state {
3838
#endif
3939
};
4040

41-
/* _PyFloat_{Pack,Unpack}{4,8}
42-
*
43-
* The struct and pickle (at least) modules need an efficient platform-
44-
* independent way to store floating-point values as byte strings.
45-
* The Pack routines produce a string from a C double, and the Unpack
46-
* routines produce a C double from such a string. The suffix (4 or 8)
47-
* specifies the number of bytes in the string.
48-
*
49-
* On platforms that appear to use (see _PyFloat_Init()) IEEE-754 formats
50-
* these functions work by copying bits. On other platforms, the formats the
51-
* 4- byte format is identical to the IEEE-754 single precision format, and
52-
* the 8-byte format to the IEEE-754 double precision format, although the
53-
* packing of INFs and NaNs (if such things exist on the platform) isn't
54-
* handled correctly, and attempting to unpack a string containing an IEEE
55-
* INF or NaN will raise an exception.
56-
*
57-
* On non-IEEE platforms with more precision, or larger dynamic range, than
58-
* 754 supports, not all values can be packed; on non-IEEE platforms with less
59-
* precision, or smaller dynamic range, not all values can be unpacked. What
60-
* happens in such cases is partly accidental (alas).
61-
*/
62-
63-
/* The pack routines write 2, 4 or 8 bytes, starting at p. le is a bool
64-
* argument, true if you want the string in little-endian format (exponent
65-
* last, at p+1, p+3 or p+7), false if you want big-endian format (exponent
66-
* first, at p).
67-
* Return value: 0 if all is OK, -1 if error (and an exception is
68-
* set, most likely OverflowError).
69-
* There are two problems on non-IEEE platforms:
70-
* 1): What this does is undefined if x is a NaN or infinity.
71-
* 2): -0.0 and +0.0 produce the same string.
72-
*/
73-
PyAPI_FUNC(int) _PyFloat_Pack2(double x, unsigned char *p, int le);
74-
PyAPI_FUNC(int) _PyFloat_Pack4(double x, unsigned char *p, int le);
75-
PyAPI_FUNC(int) _PyFloat_Pack8(double x, unsigned char *p, int le);
76-
77-
/* The unpack routines read 2, 4 or 8 bytes, starting at p. le is a bool
78-
* argument, true if the string is in little-endian format (exponent
79-
* last, at p+1, p+3 or p+7), false if big-endian (exponent first, at p).
80-
* Return value: The unpacked double. On error, this is -1.0 and
81-
* PyErr_Occurred() is true (and an exception is set, most likely
82-
* OverflowError). Note that on a non-IEEE platform this will refuse
83-
* to unpack a string that represents a NaN or infinity.
84-
*/
85-
PyAPI_FUNC(double) _PyFloat_Unpack2(const unsigned char *p, int le);
86-
PyAPI_FUNC(double) _PyFloat_Unpack4(const unsigned char *p, int le);
87-
PyAPI_FUNC(double) _PyFloat_Unpack8(const unsigned char *p, int le);
88-
8941

9042
PyAPI_FUNC(void) _PyFloat_DebugMallocStats(FILE* out);
9143

Diff for: ‎Lib/test/test_float.py

+73-1
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,14 @@
1212
from test.test_grammar import (VALID_UNDERSCORE_LITERALS,
1313
INVALID_UNDERSCORE_LITERALS)
1414
from math import isinf, isnan, copysign, ldexp
15+
import math
1516

17+
try:
18+
import _testcapi
19+
except ImportError:
20+
_testcapi = None
21+
22+
HAVE_IEEE_754 = float.__getformat__("double").startswith("IEEE")
1623
INF = float("inf")
1724
NAN = float("nan")
1825

@@ -652,8 +659,9 @@ def test_float_specials_do_unpack(self):
652659
struct.unpack(fmt, data)
653660

654661
@support.requires_IEEE_754
662+
@unittest.skipIf(_testcapi is None, 'needs _testcapi')
655663
def test_serialized_float_rounding(self):
656-
FLT_MAX = import_helper.import_module('_testcapi').FLT_MAX
664+
FLT_MAX = _testcapi.FLT_MAX
657665
self.assertEqual(struct.pack("<f", 3.40282356e38), struct.pack("<f", FLT_MAX))
658666
self.assertEqual(struct.pack("<f", -3.40282356e38), struct.pack("<f", -FLT_MAX))
659667

@@ -1488,5 +1496,69 @@ def __init__(self, value):
14881496
self.assertEqual(getattr(f, 'foo', 'none'), 'bar')
14891497

14901498

1499+
# Test PyFloat_Pack2(), PyFloat_Pack4() and PyFloat_Pack8()
1500+
# Test PyFloat_Unpack2(), PyFloat_Unpack4() and PyFloat_Unpack8()
1501+
BIG_ENDIAN = 0
1502+
LITTLE_ENDIAN = 1
1503+
EPSILON = {
1504+
2: 2.0 ** -11, # binary16
1505+
4: 2.0 ** -24, # binary32
1506+
8: 2.0 ** -53, # binary64
1507+
}
1508+
1509+
@unittest.skipIf(_testcapi is None, 'needs _testcapi')
1510+
class PackTests(unittest.TestCase):
1511+
def test_pack(self):
1512+
self.assertEqual(_testcapi.float_pack(2, 1.5, BIG_ENDIAN),
1513+
b'>\x00')
1514+
self.assertEqual(_testcapi.float_pack(4, 1.5, BIG_ENDIAN),
1515+
b'?\xc0\x00\x00')
1516+
self.assertEqual(_testcapi.float_pack(8, 1.5, BIG_ENDIAN),
1517+
b'?\xf8\x00\x00\x00\x00\x00\x00')
1518+
self.assertEqual(_testcapi.float_pack(2, 1.5, LITTLE_ENDIAN),
1519+
b'\x00>')
1520+
self.assertEqual(_testcapi.float_pack(4, 1.5, LITTLE_ENDIAN),
1521+
b'\x00\x00\xc0?')
1522+
self.assertEqual(_testcapi.float_pack(8, 1.5, LITTLE_ENDIAN),
1523+
b'\x00\x00\x00\x00\x00\x00\xf8?')
1524+
1525+
def test_unpack(self):
1526+
self.assertEqual(_testcapi.float_unpack(b'>\x00', BIG_ENDIAN),
1527+
1.5)
1528+
self.assertEqual(_testcapi.float_unpack(b'?\xc0\x00\x00', BIG_ENDIAN),
1529+
1.5)
1530+
self.assertEqual(_testcapi.float_unpack(b'?\xf8\x00\x00\x00\x00\x00\x00', BIG_ENDIAN),
1531+
1.5)
1532+
self.assertEqual(_testcapi.float_unpack(b'\x00>', LITTLE_ENDIAN),
1533+
1.5)
1534+
self.assertEqual(_testcapi.float_unpack(b'\x00\x00\xc0?', LITTLE_ENDIAN),
1535+
1.5)
1536+
self.assertEqual(_testcapi.float_unpack(b'\x00\x00\x00\x00\x00\x00\xf8?', LITTLE_ENDIAN),
1537+
1.5)
1538+
1539+
def test_roundtrip(self):
1540+
large = 2.0 ** 100
1541+
values = [1.0, 1.5, large, 1.0/7, math.pi]
1542+
if HAVE_IEEE_754:
1543+
values.extend((INF, NAN))
1544+
for value in values:
1545+
for size in (2, 4, 8,):
1546+
if size == 2 and value == large:
1547+
# too large for 16-bit float
1548+
continue
1549+
rel_tol = EPSILON[size]
1550+
for endian in (BIG_ENDIAN, LITTLE_ENDIAN):
1551+
with self.subTest(value=value, size=size, endian=endian):
1552+
data = _testcapi.float_pack(size, value, endian)
1553+
value2 = _testcapi.float_unpack(data, endian)
1554+
if isnan(value):
1555+
self.assertTrue(isnan(value2), (value, value2))
1556+
elif size < 8:
1557+
self.assertTrue(math.isclose(value2, value, rel_tol=rel_tol),
1558+
(value, value2))
1559+
else:
1560+
self.assertEqual(value2, value)
1561+
1562+
14911563
if __name__ == '__main__':
14921564
unittest.main()

Diff for: ‎Misc/NEWS.d/next/C API/2022-03-03-11-12-33.bpo-46906.-olyBI.rst

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
Add new functions to pack and unpack C double (serialize and deserialize):
2+
:c:func:`PyFloat_Pack2`, :c:func:`PyFloat_Pack4`, :c:func:`PyFloat_Pack8`,
3+
:c:func:`PyFloat_Unpack2`, :c:func:`PyFloat_Unpack4` and
4+
:c:func:`PyFloat_Unpack8`. Patch by Victor Stinner.

Diff for: ‎Modules/_ctypes/cfield.c

+8-9
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,6 @@
1010

1111
#include "pycore_bitutils.h" // _Py_bswap32()
1212
#include "pycore_call.h" // _PyObject_CallNoArgs()
13-
#include "pycore_floatobject.h" // _PyFloat_Pack8()
1413

1514
#include <ffi.h>
1615
#include "ctypes.h"
@@ -1009,10 +1008,10 @@ d_set_sw(void *ptr, PyObject *value, Py_ssize_t size)
10091008
if (x == -1 && PyErr_Occurred())
10101009
return NULL;
10111010
#ifdef WORDS_BIGENDIAN
1012-
if (_PyFloat_Pack8(x, (unsigned char *)ptr, 1))
1011+
if (PyFloat_Pack8(x, ptr, 1))
10131012
return NULL;
10141013
#else
1015-
if (_PyFloat_Pack8(x, (unsigned char *)ptr, 0))
1014+
if (PyFloat_Pack8(x, ptr, 0))
10161015
return NULL;
10171016
#endif
10181017
_RET(value);
@@ -1022,9 +1021,9 @@ static PyObject *
10221021
d_get_sw(void *ptr, Py_ssize_t size)
10231022
{
10241023
#ifdef WORDS_BIGENDIAN
1025-
return PyFloat_FromDouble(_PyFloat_Unpack8(ptr, 1));
1024+
return PyFloat_FromDouble(PyFloat_Unpack8(ptr, 1));
10261025
#else
1027-
return PyFloat_FromDouble(_PyFloat_Unpack8(ptr, 0));
1026+
return PyFloat_FromDouble(PyFloat_Unpack8(ptr, 0));
10281027
#endif
10291028
}
10301029

@@ -1057,10 +1056,10 @@ f_set_sw(void *ptr, PyObject *value, Py_ssize_t size)
10571056
if (x == -1 && PyErr_Occurred())
10581057
return NULL;
10591058
#ifdef WORDS_BIGENDIAN
1060-
if (_PyFloat_Pack4(x, (unsigned char *)ptr, 1))
1059+
if (PyFloat_Pack4(x, ptr, 1))
10611060
return NULL;
10621061
#else
1063-
if (_PyFloat_Pack4(x, (unsigned char *)ptr, 0))
1062+
if (PyFloat_Pack4(x, ptr, 0))
10641063
return NULL;
10651064
#endif
10661065
_RET(value);
@@ -1070,9 +1069,9 @@ static PyObject *
10701069
f_get_sw(void *ptr, Py_ssize_t size)
10711070
{
10721071
#ifdef WORDS_BIGENDIAN
1073-
return PyFloat_FromDouble(_PyFloat_Unpack4(ptr, 1));
1072+
return PyFloat_FromDouble(PyFloat_Unpack4(ptr, 1));
10741073
#else
1075-
return PyFloat_FromDouble(_PyFloat_Unpack4(ptr, 0));
1074+
return PyFloat_FromDouble(PyFloat_Unpack4(ptr, 0));
10761075
#endif
10771076
}
10781077

Diff for: ‎Modules/_pickle.c

+2-3
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,6 @@
99
#endif
1010

1111
#include "Python.h"
12-
#include "pycore_floatobject.h" // _PyFloat_Pack8()
1312
#include "pycore_moduleobject.h" // _PyModule_GetState()
1413
#include "pycore_runtime.h" // _Py_ID()
1514
#include "pycore_pystate.h" // _PyThreadState_GET()
@@ -2244,7 +2243,7 @@ save_float(PicklerObject *self, PyObject *obj)
22442243
if (self->bin) {
22452244
char pdata[9];
22462245
pdata[0] = BINFLOAT;
2247-
if (_PyFloat_Pack8(x, (unsigned char *)&pdata[1], 0) < 0)
2246+
if (PyFloat_Pack8(x, &pdata[1], 0) < 0)
22482247
return -1;
22492248
if (_Pickler_Write(self, pdata, 9) < 0)
22502249
return -1;
@@ -5395,7 +5394,7 @@ load_binfloat(UnpicklerObject *self)
53955394
if (_Unpickler_Read(self, &s, 8) < 0)
53965395
return -1;
53975396

5398-
x = _PyFloat_Unpack8((unsigned char *)s, 0);
5397+
x = PyFloat_Unpack8(s, 0);
53995398
if (x == -1.0 && PyErr_Occurred())
54005399
return -1;
54015400

0 commit comments

Comments
 (0)