Skip to content

FIX Allow more specific controller limits by specifying filepath, not just prefix - #278

Merged
ogrisel merged 12 commits into
joblib:masterfrom
itamarst:custom-controller-limits
Oct 27, 2026
Merged

ogrisel merged 12 commits into
joblib:masterfrom
itamarst:custom-controller-limits

Conversation

@itamarst

@itamarst itamarst commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #277

Previously prefix was used as the controlling key to identify LibControllers, but prefix is not guaranteed to be unique. Thus the switch to filepath.

While I was at it, I deleted some code that did unnecessary work of setting unused private attributes, possibly remnants of some old refactoring.

This is the companion to #275 so joblib can more reliably say "OpenBLAS+pthreads, if used, will be either 1 or cpu_count() cores and that's it". Given this change, joblib would do something like this:

def set_num_threads(num_threads):
    tp_controller = get_cached_controller()
    limits = []
    for lib_controller in tp_controller.lib_controllers:
        # filepath for threadpoolctl that has this PR, prefix for backwards compat with older versions
        limit = {"filepath": lib_controller.filepath, "prefix": lib_controller.prefix}
        if isinstance(lib_controller, OpenBLASController) and lib_controller.threading_layer != "openmp":
            cpus = cpu_count(process_wide=True)
            if num_threads >= 0.75 * cpus:
                limit["num_threads"] = 1
            else:
                limit["num_threads"] = cpus
        else:
            limit["num_threads"] = num_threads
        limits.append(limit)
    tp_controller.limit(limits=limits)

Also, remove an unnecessary attribute and correct a docstring.
@itamarst itamarst changed the title FIX More specific controller limits FIX Allow more specific controller limits by specifying filepath, not just prefix Oct 26, 2026
@itamarst
itamarst marked this pull request as ready for review October 26, 2026 17:30

@ogrisel ogrisel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM besides the following small details.

Comment thread tests/test_threadpoolctl.py Outdated
Comment thread threadpoolctl.py Outdated
itamarst and others added 2 commits October 27, 2026 07:54
Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
@itamarst

Copy link
Copy Markdown
Contributor Author

I updated the docs to cover lists and "filepath".

@ogrisel

ogrisel commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

Thanks for the follow-up. Let me merge this.

@ogrisel
ogrisel merged commit a2824c6 into joblib:master Oct 27, 2026
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

limits= dict and list form don't allow full customizaton

2 participants