Skip to content

py/objcode: Implement PEP626 to add co_lines to code objects on settrace builds. #16989

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

AJMansfield
Copy link
Contributor

Summary

This PR implements PEP 626 – Precise line numbers for debugging and other tools..

This adds one method, co_lines, that returns an iterator over (begin, end, line_no) tuples of line numbers for applicable bytecode ranges.

Testing

This PR includes automated tests of all of the currently-implemented code object attributes, including the existing attributes that previously had no test coverage.

The actual values in co_lines can't be compared directly to CPython's, but all of the invariants that the method is meant to observe are checked.

In addition to the github workflow checks, I've also manually verified that all tests pass when run against a unix MICROPY_PY_SYS_SETTRACE build. (In particular, I've checked to make sure they're not skipped).

Trade-offs and Alternatives

As in CPython, this is implemented as its own distinct iterator object. Potentially, this function could be functionally replicated by instead eagerly building a list of the tuples and returning an iterator over that list, at a slightly smaller code size; but the iterator approach allows for a tighter behavioral match with CPython (fun_code_full.py:25), and code size is not at nearly the same premium in settrace builds.

Copy link

codecov bot commented Mar 21, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.56%. Comparing base (f1018ee) to head (c27a6ad).
Report is 441 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #16989      +/-   ##
==========================================
+ Coverage   98.54%   98.56%   +0.02%     
==========================================
  Files         169      169              
  Lines       21877    21945      +68     
==========================================
+ Hits        21558    21631      +73     
+ Misses        319      314       -5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link

github-actions bot commented Mar 21, 2025

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +8 +0.001% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

@AJMansfield AJMansfield changed the title py/objcode: Implement PEP626 to add the co_lines to code objects. py/objcode: Implement PEP626 to add co_lines to code objects on settrace builds. Mar 21, 2025
@Gadgetoid
Copy link
Contributor

Related to #16797?

@dpgeorge dpgeorge added the py-core Relates to py/ directory in source label Mar 27, 2025
Copy link
Member

@dpgeorge dpgeorge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this looks like a nice addition!

I think we should then deprecate co_lnotab following CPython.

And it would be good to add another test to this PR, to test the actual numeric values from co_lines(), to make sure they match the expected bytecode offsets for a simple test function (that test would need a corresponding .py.exp file because you can't use CPython to work out the right answer).

iter->bc = 0;
iter->source_line = 1;
iter->ci = self->rc->prelude.line_info;
code_colines_next(MP_OBJ_FROM_PTR(iter));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to discard the first line-no entry, is that intended?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is intentional; this actually works around a bit of weirdness the test case helped me discover. For whatever reason, the first entry in this table always has a zero-length bytecode segment attributed to line 1.

This might be something to fix in the bytecode emitter, though that could also have knock-on effects for the other consumers of that table.

Skipping that first entry results in co_lines behavior behavior that matches CPython 3.10, where it only enumerates the body lines of a function --- though either way that's a diff with CPython 3.12, where co_lines includes the function's declaration line.

@AJMansfield
Copy link
Contributor Author

Hmm, actually, looking at the compilation result, it looks like gcc -O3 isn't quite strong enough to make the whole result.info_increment part fully transparent --- at least in ARM, there's two extra opcodes with it, associated with moving around that 1 or 2 increment value and incrementing the line_info value with it.

https://godbolt.org/z/hqoYodEaE

@AJMansfield
Copy link
Contributor Author

AJMansfield commented Jun 26, 2025

There we are, having it return the new line_info value instead of an increment value for it manages to get it fully elided: https://godbolt.org/z/bEY6ffTK8

Though, that makes it much less clear what's going on --- and there's similar differences in the other contexts with how incrementing bc and the like gets rolled in; doing the same thing for them makes it even harder and also ends up pessimizing the mp_bytecode_get_source_line to something even larger than without it.

(Also, clang gets the exact same compilation result regardless of whether or which way I refactor it.)

I'll leave the PR as-is unless we really feel like it's important to turn the screws as tight as possible just to win back 2 opcodes here.

@AJMansfield
Copy link
Contributor Author

AJMansfield commented Jun 26, 2025

The thread/thread_gc1.py test failure in unix port / settrace_stackless (pull_request) seems to be spurious.

Was initially concerned if there was some sort of overflow making mp_obj_malloc(mp_obj_colines_iter_t, &mp_type_polymorph_iter); somehow a leak (with the fact that mp_obj_colines_iter_t is a lot larger than the mp_obj_iter_buf_t type used for this in mp_obj_tuple_getiter) --- but that test doesn't even interact with code objects, so idk.

py/bc.h Outdated
size_t info_increment;
} mp_code_lineinfo_t;

static inline mp_code_lineinfo_t mp_bytecode_decode_lineinfo(const byte *line_info) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code size increases with this bit of logic factored out into this inline function.

I had a little play, and it seems that code size can be reclaimed by passing in a pointer to line_info, updating it here as needed (adding 1 or 2) and not returning info_increment. Ie:

 static inline mp_code_lineinfo_t mp_bytecode_decode_lineinfo(const byte **line_info) {
    mp_code_lineinfo_t result;
    size_t c = **line_info;
    if (...) {
        ...
        *line_info += 1;
    } else {
        ...
        *line_info += 2;
    }
    return result;
}

I suggest making that change to try and get the code size diff to be +0.

Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
@AJMansfield AJMansfield force-pushed the code_co_lines branch 3 times, most recently from 0acb0e6 to 5c31dc4 Compare July 1, 2025 17:16
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
@AJMansfield
Copy link
Contributor Author

AJMansfield commented Jul 1, 2025

And it would be good to add another test to this PR, to test the actual numeric values from co_lines(), to make sure they match the expected bytecode offsets for a simple test function (that test would need a corresponding .py.exp file because you can't use CPython to work out the right answer).

I've been playing around with these tests; at the moment the test I added also covers the cases where a correct co_lines implementation might need to sum together multiple successive table entries to get the correct answer --- which it currently doesn't do.

The current outputs from co_lines are still meaningful and correct in some sense, though --- is that summing behavior actually something this needs to implement? Or can these extra coverage cases just be dropped (or made into cpydiffs or something)?

Also, should I add coverage for a case where the table needs to skip more than 2048 lines at once? (e.g. to overflow the 11-bit line number increment in the 2-byte format, and require a multi-entry encoding for it like bc+0, line+2047; bc+4, line+12)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
py-core Relates to py/ directory in source
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants