Skip to content

fix(charts): pandas 3 fixes for box plot counts, pivot CSV totals and calendar labels - #45865

Open
hughhhh wants to merge 2 commits into
masterfrom
fix/pandas3-chart-smoke-findings
Open

hughhhh wants to merge 2 commits into
masterfrom
fix/pandas3-chart-smoke-findings

Conversation

@hughhhh

@hughhhh hughhhh commented Oct 29, 2026

Copy link
Copy Markdown
Member

SUMMARY

Fixes three issues from a pandas 3.0 chart smoke test:

  1. Box plot __count counted NULL values. np.ma.count on a plain array counts every element, so rows whose metric is NULL were counted, and a group with only NULLs counted its row total. This happens on pandas 2 as well. The count uses Series.count, which leaves NULLs out.
  2. Pivot table CSV export with totals showed a blank instead of nan. In pivot_df, df[col].replace("SUPERSET_PANDAS_NAN", ..., inplace=True) is a chained inplace call. Under pandas 3 copy-on-write it only updates a copy (pandas raises ChainedAssignmentError for it), so the "nan" text for missing cells in non-numeric metric columns never reached the frame. The later frame-wide replace turned those cells into np.nan, which a CSV writes as empty. The result is now assigned back to the column. Totals values were already correct.
  3. Calendar heatmap with "show values" showed null on days with no data. cal-heatmap passes null for those days, and the number formatter renders it as "null". The label is empty for missing values.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

Not captured. The calendar change only affects days without data when "Show Values" is on: before, the cell read null; after, it is empty.

TESTING INSTRUCTIONS

  • Box plot: proved by test_boxplot_count_excludes_null_metric_values. It fails on master (counts {a: 3, b: 2} instead of {a: 2, b: 0}) under both pandas 2.3.3 and 3.0.6, and passes with the fix.
  • Pivot CSV totals: proved by test_pivot_df_rows_total_keeps_nan_text_in_non_numeric_metric. It fails on master under pandas 3.0.6 (nan == 'nan') and passes with the fix. tests/unit_tests/charts/ also passes with -W error::pandas.errors.ChainedAssignmentError.
  • Calendar labels: proved by the new Calendar value labels leave days without data blank test in ReactCalendar.test.tsx. It fails on master ("null") and passes with the fix.
pytest tests/unit_tests/pandas_postprocessing/test_boxplot.py tests/unit_tests/charts/test_client_processing.py
cd superset-frontend && npx jest plugins/plugin-chart-calendar

Manual check: build a calendar heatmap with "Show Values" on over a range with gaps. Empty days should carry no label.

ADDITIONAL INFORMATION

  • Has associated issue:
  • Required feature flags:
  • Changes UI
  • Includes DB Migration (follow approval process in SIP-59)
    • Migration is atomic, supports rollback & is backwards-compatible
    • Confirm DB migration upgrade and downgrade tested
    • Runtime estimates and downtime expectations provided
  • Introduces new feature or API
  • Removes existing feature or API

🤖 Generated with Claude Code

… calendar labels

- Box plot: count observations with Series.count so NULL metric values are
  left out of `__count` (np.ma.count counted every row).
- Pivot table totals: assign the placeholder replacement back to the column;
  under copy-on-write the chained inplace replace never reached the frame, so
  a missing non-numeric cell exported as a blank instead of "nan".
- Calendar heatmap: leave the value label empty on days without data rather
  than rendering a literal "null".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.71%. Comparing base (6fced4f) to head (24e2daa).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master   #45865   +/-   ##
=======================================
  Coverage   84.70%   84.71%           
=======================================
  Files        3069     3069           
  Lines      205649   205663   +14     
  Branches    48335    48336    +1     
=======================================
+ Hits       174200   174222   +22     
+ Misses      28156    28148    -8     
  Partials     3293     3293           
Flag Coverage Δ
hive 34.51% <0.00%> (-0.01%) ⬇️
javascript 80.95% <100.00%> (+0.01%) ⬆️
mysql 52.58% <0.00%> (+<0.01%) ⬆️
postgres 52.59% <0.00%> (-0.01%) ⬇️
presto 36.24% <0.00%> (-0.01%) ⬇️
python 87.75% <100.00%> (+<0.01%) ⬆️
sqlite 52.35% <0.00%> (-0.01%) ⬇️
unit 82.48% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@EnxDev EnxDev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read through this one and the three fixes hold up. The tests reproduce each bug, and CI is green.

One optional note inline about the same count bug in the shared aggregate map.

whisker_high = np.max
whisker_low = np.min

def count(series: Series) -> int:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit, take it or leave it. NUMPY_FUNCTIONS["count"] in pandas_postprocessing/utils.py:35 is still np.ma.count, so an aggregate or pivot step with operator: "count" keeps counting NULL rows the same way.

Same swap would fix it there too. Fine as a follow-up if you'd rather keep this PR tight.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

2 participants