Skip to content

[Bug]: Overflow of 16-bit integer in Agg renderer causes PolyCollections to be drawn at incorrect locations #23826

Closed
@Exploder98

Description

@Exploder98

Bug summary

Because the pixel x coordinates of scanline cells are casted from 32-bit integers to 16-bit ones in Agg (this line), some polygons that are outside the drawing area may actually be drawn at an incorrect location if they are far enough from the viewport so that the 16-bit integer used for actual drawing overflows and the value happens to be inside the drawing area.

Code for reproduction

import matplotlib.pyplot as plt
import matplotlib
matplotlib.use('TkAgg')

# Create a 120x120 px figure
fig = plt.figure(figsize=(120, 120), dpi=1)
# Add axis that fills the whole figure
ax = fig.add_axes([0, 0, 1, 1])
ax.axis('off')
# Plot one fill_between at x, y = 0, width = 100 and height = 50
ax.fill_between([0, 100], 0, 50, color='r')
# Plot another fill_between at x = 65536, y = 50, width=100 and height = 50
# 65536 causes the 16-bit integer to overflow so that the coordinate becomes actually 0
ax.fill_between([2 ** 16, 2 ** 16 + 100], 50, 100, color='b')
# Set limits to ensure that one pixel in the rasterized image corresponds to one unit in the axis
ax.set_xlim(-10, 110)
ax.set_ylim(-10, 110)
# Agg has the bug, Pdf does not
plt.savefig('bug.png', dpi=1)
plt.savefig('nobug.pdf', dpi=1)
plt.show()

Actual outcome

The PolyCollections are both drawn in the image, even though they are 65536 units apart and the x axis goes from -10 to 110.
bug

Expected outcome

There should only be the red rectangle, as in this PDF.
nobug.pdf

Additional information

I debugged this quite a bit, and turns out it is a bit tricky to replicate this. The PolyCollections need to be able to fit in the axes completely so that the optimized draw_markers function is called. Then, inside Agg, transforms are applied so that pixel coordinates are calculated. These coordinates are fine, as they use 32-bit integers. However, when Agg draws the scanlines, it converts the coordinates to 16-bit integers. If the axis is zoomed in enough and everything lines up just right, some points that have very high positive or very high negative pixel coordinates may appear in the viewport because the int16 conversion causes the 16-bit coordinate to over- or underflow and to be inside the viewport. An obvious fix could be to use int32 instead of int16 while drawing, but I don't know if that's possible or if matplotlib is even the right place to report, as this bug seems to be in Agg code. The original figure where I discovered this had many fill_betweens, and when zoomed in, some fill_betweens from earlier or later in the data would appear.

Note that at least on my machine, the figure that is shown in the window might not have the bug present, but the saved PNG file should have it.

I checked also the main branch (version 3.6.0.dev3132+gf8cf0ee5e7), and it has the bug.

Operating system

Probably anything where Agg runs (verified on RHEL8 and Windows)

Matplotlib Version

3.5.3

Matplotlib Backend

QtAgg

Python version

3.9.13

Jupyter version

No response

Installation

conda

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions