JobQueueCluster with local worker(s) #596

Activity

1. jacobtomlinson commented on Nov 7, 2022

   

   jacobtomlinson

   on Nov 7, 2022

   Member

   More actions

   This is an interesting question.

   I'm not sure that having the FooCluster object also spawn a local worker is the right approach here. It feels messy and a bad separation of concerns.

   Howerver I agree manually creating a local worker right now isn't the best user experience. I had a quick go and it was something like this:

   import asyncio
   
   # Using LocalCluster to sake of demo but could be any FooCluster
   from dask.distributed import LocalCluster, Nanny, Client
   from distributed.comm import get_address_host_port
   
   async def main():
       async with LocalCluster() as cluster:
           async with Nanny(*get_address_host_port(cluster.scheduler_address), silence_logs=True) as nanny:
               async with Client(cluster) as client:
                   print(cluster)
   
   if __name__ == "__main__":
       loop = asyncio.get_event_loop()
       loop.run_until_complete(main())

   It would be much nicer if there was some non-asyncio way of starting a worker by simply passing it the cluster object, the same way we can create a client from a cluster today.

   Something like this:

   from dask.distributed import LocalCluster, Nanny, Client
   
   cluster = LocalCluster()
   local_worker = Nanny(cluster)
   client = Client(cluster)
2. guillaumeeb commented on Nov 10, 2022

   

   guillaumeeb

   on Nov 10, 2022

   Member

   More actions

   Hi @fnattino, thanks for raising this. This is a problem we already identified but with no good solutions yet. There is some discussion about it in some issues, (here for example) and a solution (yet to be implemented) is proposed in #419.

   Basically, the idea I had is to be able to create a LocalCluster and reuse its Scheduler object to build a JobQueueCluster on top of it.

   But I also like the proposal of @jacobtomlinson!

   @fnattino if you'd like to give a try implementing one of these solutions, this would be very much appreciated :).
3. fnattino commented on Nov 14, 2022

   

   fnattino

   on Nov 14, 2022

   ContributorAuthor

   More actions

   @jacobtomlinson @guillaumeeb thank you both for your quick replies and for pointing out other relevant issues! I will try to experiment a bit with your suggestions and, should I get to something useful, I will keep you posted. Thanks again!