py: Add enum support and minimal metaclass features #18416
Draft
+15,111
−62
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Code size report: |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #18416 +/- ##
==========================================
- Coverage 98.38% 98.23% -0.15%
==========================================
Files 171 171
Lines 22294 22455 +161
==========================================
+ Hits 21933 22058 +125
- Misses 361 397 +36 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Contributor
|
Thanks Andrew, |
Contributor
Author
|
Thanks @Josverl I did review / refer to yours but was lucky enough to have an older initial commit from @dpgeorge to start from so tried to keep to its style and make this as minimal as possible while still enabling Enum (as my primary use case for metaclasses). It was only while working on this @mattytrentini happened to mention python-statemachine for another project which uses some metaclass stuff so aimed for compatibility with that too |
andrewleech
force-pushed
the
py-metaclass-enum
branch
3 times, most recently
from
5838d45 to
ca58524
Compare
andrewleech
force-pushed
the
py-metaclass-enum
branch
from
4d81f4a to
114b227
Compare
Signed-off-by: Damien George <damien@micropython.org>
Adds four new configuration flags to control metaclass functionality: - MICROPY_PY_METACLASS_INIT: Enable metaclass __init__ invocation - MICROPY_PY_METACLASS_OPS: Enable metaclass operator overloading - MICROPY_PY_METACLASS_PROPERTIES: Enable metaclass properties/methods - MICROPY_PY_METACLASS_PREPARE: Enable __prepare__ method (PEP 3115) These flags allow ports to balance functionality against code size, enabling enum support and python-statemachine compatibility. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Fixes type checking throughout the codebase to support custom metaclasses: - py/vm: Fix LOAD_ATTR fast path to avoid treating type objects as instances - py/objobject: Update object.__new__ to accept custom metaclasses - py/objtype: Replace assert() checks with mp_obj_is_subclass_fast() Adds metaclass method support: - type_make_new now looks up custom __new__ through inheritance chain - type_call checks for custom __call__ on metaclass - Enables metaclass customization of class creation and instantiation Removes DEBUG_printf statements for cleaner production code. This enables enum.Enum and python-statemachine compatibility. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Implements PEP 3115 by calling __prepare__ before executing class body. The __prepare__ method allows metaclasses to return a custom namespace dict (e.g., OrderedDict) for tracking member insertion order. Execution order in __build_class__: 1. Determine metaclass 2. Call __prepare__(name, bases) if it exists 3. Execute class body in returned namespace 4. Call metaclass(name, bases, namespace) to create class This is required for enum.auto() to generate sequential values based on definition order. Guarded by MICROPY_PY_METACLASS_PREPARE. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Metaclass tests: - class_metaclass_init.py: Test __init__ invocation on metaclass - class_metaclass_prepare.py: Test __prepare__ method (PEP 3115) - class_metaclass_property.py: Test properties and methods on metaclasses Enum tests: - enum_auto.py: Test auto() value generation - enum_flag.py: Test Flag and IntFlag bitwise operations - enum_strenum.py: Test StrEnum string-valued enums CPython difference: - types_enum_isinstance.py: Document isinstance() behavior difference These tests validate the metaclass features added in previous commits and provide compatibility verification with micropython-lib enum package. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Adds enum package from micropython-lib to unix port variants: - coverage variant: For testing metaclass and enum features - standard variant: For general enum availability The enum package provides Enum, IntEnum, Flag, IntFlag, StrEnum, and auto() compatible with CPython's enum module. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Updates micropython-lib submodule to include enum package with fixes: - Fix __import__() to use positional args (MicroPython compatibility) - Fix IntFlag bitwise operations to use _value_ attribute directly - Add Flag, IntFlag, StrEnum, auto(), and unique() support These changes enable full enum functionality with the metaclass features added in previous commits. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
The metaclass.py test now produces output because metaclass __init__ invocation is now implemented and functional. Generate the expected output file based on the correct behavior. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Adds feature detection to skip tests when required features are not available (e.g. in minimal/standard build variants). - class_metaclass_init.py: Skip if MICROPY_PY_METACLASS_INIT disabled - class_metaclass_property.py: Skip if MICROPY_PY_METACLASS_PROPERTIES disabled - class_metaclass_prepare.py: Already had skip for __prepare__ - enum_auto.py: Add enum import check before __prepare__ check - enum_flag.py: Skip if enum or __prepare__ not available - enum_strenum.py: Skip if enum module not available - metaclass.py: Skip if MICROPY_PY_METACLASS_INIT disabled This fixes CI failures on build variants where these features are disabled by config. Signed-off-by: Andrew Leech <andrew@alelec.net>
…lass. When super().__init__() was called from within a custom metaclass __init__, the super_attr() function incorrectly mapped __init__ lookups to the make_new slot for all objects. For type objects, this caused type_make_new() to be invoked, which called mp_obj_new_type() again, creating a duplicate type object that overwrote the original's memory. On 32-bit architectures, this memory corruption manifested as corrupted slot_index fields, causing assertion failures when accessing locals_dict. The fix: - Add type.__init__() as a no-op that accepts standard metaclass arguments - In super_attr(), detect when super().__init__() is called on a type object (metaclass context) and return type.__init__() instead of mapping to make_new - Only map __init__ to make_new for regular instance initialization This preserves the existing behavior for regular instances while fixing metaclass super().__init__() calls to not recreate type objects. Fixes: #XXXXX Signed-off-by: Corona <corona@example.com> Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
When creating a subclass of type (e.g., class D(type)), we inherit slots
from the base type. However, some slots like unary_op and binary_op are
only defined in mp_type_type when MICROPY_PY_METACLASS_OPS is enabled
(at EXTRA_FEATURES level or higher).
Previously, MP_OBJ_TYPE_SET_SLOT would unconditionally set slot_index
even when the slot value was NULL. This caused MP_OBJ_TYPE_HAS_SLOT
to return true for NULL slots, leading to NULL function pointer calls
when using operators on type objects in minimal variant.
Fix by adding MP_OBJ_TYPE_SET_SLOT_IF_EXISTS macro that only sets the
slot index if the value is non-NULL, and use it when inheriting slots
from a type base class.
Fixes crash in basics/unary_op.py on minimal variant:
class D(type): pass
d = D('foo', (), {})
print(not d) # crashed before fix
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
The previous implementation used a do-while(0) pattern which MSVC doesn't accept. Replace with a conditional expression using the comma operator that works with both GCC and MSVC compilers. This fixes Windows port compilation error C2121. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
MSVC doesn't support preprocessor conditionals inside macro arguments. Split the mp_type_type definition into two separate invocations based on MICROPY_PY_METACLASS_OPS to avoid placing #if/#endif inside the macro call. This fixes Windows port MSVC compilation errors C2121 and C2059. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
- Remove unnecessary True comparisons (use truthiness) - Remove unused variable assignment - Remove f-string prefix where not needed Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
andrewleech
force-pushed
the
py-metaclass-enum
branch
from
114b227 to
88016b1
Compare
…iginal. Refactor to preserve dpgeorge's original design pattern: - Move type___new__ and type___init__ declarations to file header - Simplify type_make_new to just dispatch to type___new__ - Simplify type_call to use mp_load_method_maybe instead of mp_obj_class_lookup - Reduce code complexity while maintaining all functionality Binary size reduced by 128 bytes. All tests pass. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Adds enum support (Enum, IntEnum, Flag, IntFlag, StrEnum, auto, unique) via micropython-lib submodule and implements minimal metaclass features needed to support it and python-statemachine.
Built on dpgeorge's b31c1de metaclass branch from 2020. Reviewed Jos Verlinde's PR #18362 (PEP 3115/487 metaclass support), then focused on the subset needed for enum and python-statemachine without implementing full PEP 487 init_subclass.
Metaclass Features
Implemented as optional ROM-level features with separate config flags:
__init__invocationMICROPY_PY_METACLASS_INITMICROPY_PY_METACLASS_OPSMICROPY_PY_METACLASS_PROPERTIES__prepare__(PEP 3115)MICROPY_PY_METACLASS_PREPARE__init_subclass__(PEP 487)Total C overhead: 540 bytes when all features enabled (FULL level).
The init feature enables python-statemachine's class registration pattern. Properties enable accessing
.eventsand.stateson the class. Operators enablelen(EnumClass)andmember in EnumClass. prepare enables enum's auto() value generation.Enum Features
Complete implementation via micropython-lib submodule, based on PEP 435 (basic enums) and PEP 663 (Flag additions):
Frozen as bytecode: ~5,428 bytes.
Modular structure with lazy loading:
Total implementation: 540 bytes C + 5,428 bytes Python = 5,968 bytes (1.6% increase on STM32 PYBV10).
CPython Compatibility
Tested against CPython 3.13's official enum test suite:
Works:
Not implemented:
Enum('Name', 'A B C')) - use class syntax insteadKnown limitation: IntEnum members fail isinstance(member, int) check but all operations work correctly. Documented in tests/cpydiff/types_enum_isinstance.py.
STM32 Size Measurements (PYBV10)
Individual feature costs:
Cumulative by ROM level:
With enum module frozen:
Note: cumulative cost (540 bytes) is less than sum of individual features (548 bytes) due to code sharing.
Testing
Unix port coverage variant:
Tests added:
Tests fixed:
Implementation Details
Core C files modified:
Enum module:
The enum implementation lives in micropython-lib (separate repository) and is included via submodule reference. Both Unix variants (standard and coverage) freeze the enum package into their builds.