Skip to content

fix: support grouped null_count on pandas-like and PyArrow - #4102

Open
dsilvadeepal wants to merge 1 commit into
narwhals-dev:mainfrom
dsilvadeepal:fix/grouped-null-count-2484
Open

dsilvadeepal wants to merge 1 commit into
narwhals-dev:mainfrom
dsilvadeepal:fix/grouped-null-count-2484

Conversation

@dsilvadeepal

@dsilvadeepal dsilvadeepal commented Oct 22, 2026 •

Copy link
Copy Markdown

Description

Fixes #2484.

Grouped null_count() previously used the slow pandas apply fallback and failed on PyArrow. Use group size minus valid count on pandas-like backends and Arrow's native null-count aggregation. Keep existing window behavior unchanged.

Regression tests cover aliases, multiple columns, mixed/all-null/no-null groups, null group keys, empty inputs, and input preservation.

Validation: 17,460 tests passed with 100% coverage locally (CI-only version assertion enabled), 486 doctests passed, all three type checkers and repository hooks passed, and the strict documentation build passed. GPU/cuDF and Modin were not tested.

AI assistance: Codex — sol/astra/luna.

Local test screenshot: Screenshot 2026-10-22 at 11.56.17 AM.png (include with the PR description).

What type of PR is this? (check all applicable)

  • 💾 Refactor
  • ✨ Feature
  • 🐛 Bug Fix
  • 🔧 Optimization
  • 📝 Documentation
  • ✅ Test
  • 🐳 Other

Related issues

AI assistance

  • No AI tools were used for this PR.
  • AI tools were used.
    AI assistance: Codex — sol/astra/luna.

Checklist

  • Code follows style guide (ruff)

  • Tests added

  • Documented the changes

  • If this is your first PR to narwhals, attach a screenshot of pytest passing locally (not CI):

    PYTEST_ADDOPTS="--numprocesses=logical" \
    make run-ci DEPS="--extra pandas --extra dask --group core-tests --group sklearn --group plugins" \
    CMD="pytest tests --cov=src --cov=tests --runslow --constructors=pandas,pandas[nullable],pandas[pyarrow],pyarrow,polars[eager],polars[lazy],dask,duckdb,sqlframe"
Screenshot 2026-10-22 at 11 56 17 AM

@coderabbitai

coderabbitai Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: d991eb64-c8eb-416b-8ea7-32498358d6fb
📥 Commits

Reviewing files that changed from the base of the PR and between 1b7d974 and 7d56c18.

📒 Files selected for processing: 5
  • docs/concepts/improve_group_by_operation.md
  • src/narwhals/_arrow/group_by.py
  • src/narwhals/_compliant/typing.py
  • src/narwhals/_pandas_like/group_by.py
  • tests/frame/group_by_test.py

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.

📝 Walkthrough

Grouped null_count aggregations are implemented for pandas-like and PyArrow backends. Tests cover multiple columns, null grouping keys, and empty inputs. Documentation describes the backend implementations.

Changes: Grouped null-count aggregation

Layer / File(s) Summary
Implement and validate grouped null counts
src/narwhals/_compliant/typing.py, src/narwhals/_pandas_like/group_by.py, src/narwhals/_arrow/group_by.py, tests/frame/group_by_test.py, docs/concepts/improve_group_by_operation.md
The aggregation type includes null_count. The pandas-like implementation subtracts grouped non-null counts from group sizes, while the PyArrow implementation uses a null-only count option. Tests cover grouped results, null-key handling, input preservation, and empty inputs. The documentation describes these implementations.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested reviewers: fbruzzesi, dangotbanned, marcogorelli

Merge Risk: ⚪ Minimal · up to 7d56c

Grouped null counts appear ready to merge after normal checks; no concrete blocking issue remains.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 8 functions across 4 files. (1 skipped: 1… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Title check Passed The title clearly and concisely describes the main change: adding grouped null_count support for pandas-like and PyArrow backends.
Linked Issues check Passed Issue #2484 requires grouped null_count() support on pandas-like backends. src/narwhals/_pandas_like/group_by.py treats elementary null_count expressions as simple and computes group size minus …
Out of Scope Changes check Passed The changed source, typing, documentation, and tests support grouped null_count(). The PyArrow implementation is also part of the current PR intent and uses the related native aggregation path. The …
Description check Passed The pull request description is complete and follows the repository template. It explains the fix, identifies the related issue, marks the PR type, discloses AI assistance, documents validation, lists…
Full details: Linked Issues check

Explanation

Issue #2484 requires grouped null_count() support on pandas-like backends. src/narwhals/_pandas_like/group_by.py treats elementary null_count expressions as simple and computes group size minus the grouped non-null count. This avoids the apply fallback. Regression tests cover multiple columns, aliases, null keys, empty inputs, and input preservation. The issue does not define a Dask-specific requirement; the tests record Dask's known limitation as an expected failure. The PR also adds native PyArrow null counting and updates the aggregation typing.

Full details: Out of Scope Changes check

Explanation

The changed source, typing, documentation, and tests support grouped null_count(). The PyArrow implementation is also part of the current PR intent and uses the related native aggregation path. The reviewed summary shows no unrelated change.

Full details: Description check

Explanation

The pull request description is complete and follows the repository template. It explains the fix, identifies the related issue, marks the PR type, discloses AI assistance, documents validation, lists testing limitations, and includes the required checklist and screenshot.

Full details: Docstring Coverage

Explanation

Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 8 functions across 4 files. (1 skipped: 1 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

enh: implement group_by(...).agg(nw.col('a').null_count()) for pandas_like

1 participant