network mode "custom_network" not supported by buildkit

related DOCKER_BUILDKIT=1 prevents custom networks using docker build --network moby/buildkit#978

Background:
Running a simple integration test fails with network option:

docker network create custom_network
docker run -d --network custom_network --name mongo mongo:3.6
docker buildx build --network custom_network --target=test .

Output:
network mode "custom_network" not supported by buildkit

Still not supported? Code related:
https://github.com/docker/buildx/blob/master/build/build.go#L462-L463

Sorry, I'm not sure if we will ever start supporting this as it makes the build dependant on the configuration of a specific node and limits the build to a single node.

That horse has bolted - SSH mount makes the build dependent upon the configuration of a single node

No, it does not. You can forward your ssh agent against any node or a cluster of nodes in buildx. Not really different than just using private images.

why would someone forward it across an insecure connection?

Why would that connection be insecure? Forwarding agent is more secure than build secrets because your nodes never get access to your keys.

if you can do all your tests in a docker build container it's one less thing to lock down.
along with the means to temporary mount secrets

We have solutions for build secrets, privileged execution modes (where you needed docker run before for more complicated integration tests) and persistent cache for your apt/npm cache etc. moby/buildkit#1337 is implementing sidecar containers support. None of this breaks the portability of the build. And if you really want it, host networking is available for you.

you might have been referring to something else.

https://medium.com/@tonistiigi/build-secrets-and-ssh-forwarding-in-docker-18-09-ae8161d066

The sidecar also sounds great and very clever and well structured. But again, 3 years ago I could build with secrets and talk to network services to run integration tests.

Also, it does work in compose while build secrets does not.

Adding another use case where specifying the network would be useful: "hermetic builds".

I'm defining a docker network with --internal that has one other container on the network, a proxy that is providing all the external libraries and files needed for the build. I'd like the docker build to run on this network without access to the external internet, but with access to that proxy.

I can do this with the classic docker build today, or I can create an entire VM with the appropriate network settings, perhaps it would also work if I setup a DinD instance, but it would be useful for buildkit to support this natively.

@bryanhuntesl The proxy vars are still supported. For this use case, cache mounts might be a better solution now https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/syntax.md#run---mounttypecache

@TBBle

@TBBle Thanks; that's also what I found. I appreciate the suggestions but don't yet have the mental energy earmarked in order to complete any. I'll wait a little longer in hopes that someone else streamlines a solution before attempting again :)

My two cents about use cases.

I was led here after trying to collapse a complex CI process into a Dockerfile. Specifically, a step runs migrations on a DB, another step dumps the DB's schema to a file, and a third step generates templates using that output. All this work follows the mandate of Docker build recipes—repeatable, portable—but the implementation of spinning up a DB doesn't fit with the paradigm.

Reason being: while migrations can run on the DB in step one, the DB itself must be running and accessible for step two, which has to run on a different image. By design, you can't keep a build layer alive once you've moved on to another one. Thus, in my use case, it points to a reasonable need for the DB to be attached as an external service, but still scoped and isolated to the build process.

Regrettably, the current tooling ecosystem almost serves this purpose. It provides a network parameter, but won't allow custom networks. There is a workaround using buildx containers, but in my experimentation, that workaround did not work-around (code not available; I could not reach the container that should allegedly have been attached).

Seeking support for the issue, one then reaches this labyrinth of support tickets. And the first thing one sees there, confusingly, is the proverbial “slap of the wrist” for wanting or expecting the feature to exist in the first place.

As I see it: Docker builds are supposed to be able to network. We know this. To avoid unpredictable build outputs, the user should use the internet only to download resources that are themselves invariant for a given set of build parameters. If the user breaks this rule, they have earned the consequences.

Why should resources isolated to a custom network be any different? Docker networks are an extension of the general concept of networking, and in this context an irreducible one. One of the tolls you pay for moving off of bare metal is that containers need a mediating layer if they want to be able to talk to each other in an un-ambiguous way. It's natural to expect that tool to be available wherever it is needed, and build-time is one of the places where it is sometimes needed.

What I’m saying is: this isn’t really about repeatability or portability, is it? It’s about difficulty of implementation. I appreciate that it takes a vast amount of work to satisfy a particular use-case without breaking something else, but that makes it all the more unfortunate to see effort be wasted on framing a difficult use-case as somehow un-“Dockerly”. Especially because it is only difficult as a result of competing priorities taking precedence (BuildKit cache optimizations etc) and especially because of CLI, docs, and GitHub leading developers down a garden path that sometimes ends at that extremely unsatisfying answer.

It's not merely difficulty of implementation; if that were the case, someone would have done it by now. I probably would have done it by now if it was merely "hard to implement".

The driving change is that with the introduction of BuildKit and buildx, the structure of the container build system has shifted to be much more strongly isolated, both from outside influences and particularly from Docker.

The build system is no longer Docker-specific. This has been part of Docker's direction for years now, as seen by the break-out of moby/moby, OCI standardisation of container formats, etc.

The upshot of that for our purposes is that Docker's internal overlay network implementation is not visible to BuildKit (docker buildx build), unless you specifically create a builder instance on that custom network (docker buildx create --driver docker-container --driver-opt "network=my-custom-network"), in the same way if you want to give your builds access to some Kubernetes environment, you create a builder instance using the Kubernetes driver. Even the buildkit instance compiled into Docker doesn't have full access to the repertoire of Docker features, as described in the last segment of #175 (comment).

So wanting to have your builds run against a custom network is not being rejected as "Possible but unDockerly". However, to do that in the current state, you actually need to do it, Docker is no longer in a position to make that work magically for you.

So "using buildx containers" is not a work-around, that's the way to achieve what you want. As it's the way to achieve anything you want from buildx that isn't available via the current built-in, limited, (and default) buildkit integration.

There's definitely space to improve the UX on "using buildx containers", but it's still going to be the same rough flow of "To do something that we can't do from the built-in buildkit builder, create a custom builder that supports what you need, use it, and tear it down when done" with varying levels of automation around it. Docker and buildx build directly probably can't do much about this, but orchestration tools like buildx bake or docker compose probably have scope to wrap this sort of thing up in their configuration files. (Although right now, neither actually offers to manage services used during build at all, so that's a larger discussion to be had).

@MattiasMartens

@MattiasMartens Very well said. I would just add that it was so easy and simple previously with the classic builder. You don't know what you'd be missing until it is deprecated! I must say I don't follow @TBBle's thought process as well as Mattias's. First, because I don't consider it "magic" to make a network available to a container. And second because "the structure of the container build system has shifted to be much more strongly isolated" - meaning isolated from what exactly? Clearly not the internet. I'm not claiming I'm owed a more streamlined compose experience, but I do feel it is important to correctly communicate the experience - as Mattias has. This design short circuits the UX of docker compose for users that run containers that need access to other containers or networks at build time. What happens, for example, when docker detects that a newer version of the container should be running from compose up and attempts to build it? Then the build will fail because compose up cannot exercise the requisite code path(s), right?

@TBBle

@TBBle If I understand correctly, you’re saying it’s not merely “hard to implement” but impossible to implement because of a decision about separation of concerns made by BuildKit?

I think I follow your explanation broadly but there are some unfortunate choices of phrase... For example "Docker is no longer in a position to make [custom networks in the build environment] work magically for you" is a strange way to phrase "Docker no longer allows one feature, provided by Docker, through the CLI, to leverage another feature, provided by Docker, through the CLI". If that’s "magic" then we are all wizards!

I would be much happier with the status quo if someone, in the five years of this issue being open, had reported success in binding their custom network to their build environment by leveraging a custom builder. So far everyone who has tried this, including myself, has reported failure. Consider this an open request for a working example.

@MattiasMartens

@MattiasMartens

Specifically, a step runs migrations on a DB, another step dumps the DB's schema to a file, and a third step generates templates using that output. All this work follows the mandate of Docker build recipes—repeatable, portable—but the implementation of spinning up a DB doesn't fit with the paradigm.

Reason being: while migrations can run on the DB in step one, the DB itself must be running and accessible for step two, which has to run on a different image. By design, you can't keep a build layer alive once you've moved on to another one. Thus, in my use case, it points to a reasonable need for the DB to be attached as an external service, but still scoped and isolated to the build process.

This application design is strongly at odds with the general best practices of microservices design and the reason that there is so much friction attempting to implement it within Docker. It implies that the state of the packaged application also includes state in an external and independent database. That tight linkage of state creates issues when upgrading and downgrading container images independent of the database.

Specifically for this issue, I'd propose that the DB should not be needed to convert migration steps into a database schema file, and that tooling that could convert between the two would significantly improve the build process for that application. If that's feasibly impossible, then starting a database, running the migration, and dumping the schema, could be done in a single step that runs the database on localhost in the background, eliminating the need for a network.

I get that this is counter to the design of some popular application frameworks. The difficulty is they are directly opposed to other efforts in the software development ecosystem, like the 12 Factor App, and now the desire to provide hermetic and reproducible builds.

I would be much happier with the status quo if someone, in the five years of this issue being open, had reported success in binding their custom network to their build environment by leveraging a custom builder. So far everyone who has tried this, including myself, has reported failure. Consider this an open request for a working example.

I posted exactly that back in January in this very issue. What was missing from my example?

I posted exactly that back in January in this very issue. What was missing from my example?

From a distance, your comment from January of this year (#175 (comment)) could be read as a demonstration of the problem, not of a solution.

On a closer read, it does explain my issue. If my priorities were such that the ability to use custom networks in the build process were worth the cruft of putting docker container inspect test-server --format "{{ (index .NetworkSettings.Networks \"custom-network\").IPAddress }}" or other such incantation in my script, I would do it.

Maybe if this pattern gets wrapped into a Docker Compose plugin, that plugin could also do something like add the container's name and IP to /etc/hosts and take most of the pain out of the process.

It implies that the state of the packaged application also includes state in an external and independent database.

In this case the starting point is a DB image. The only reason I can't apply migrations within the build script is because the DB image and the migration script image (which expects to be able to connect to a DB over the network) are two separate images, and to coordinate between them in the same dockerfile would require inter-process communication between build stages. Hence instantiating the DB on a custom network as an attempted workaround.

However, the migration script image is really just a wrapper around a binary, so I’m realizing my best bet is to copy that binary into my DB image and run it there which cuts out the need for an external network. Good riddance!

@dm17

@dm17

And second because "the structure of the container build system has shifted to be much more strongly isolated" - meaning isolated from what exactly?

Per rest of my quoted sentence, "both from outside influences and particularly from Docker.".

Specifically, the build engine is no longer part of the Docker project, in the same way the container engine stopped being part of Docker and became containerd. This allows it to be useful to and contributed-to by a wider variety of projects and be able to be used in a wider variety of environments.

Speculatively, Docker saw the writing on the wall that things like Kubernetes were not going to keep Docker underlying them forever, so anything that required Docker (like the class Docker builder) was going to fall by the wayside. You can see lots of attempts to build non-Docker container build projects, but having tried a fair few back in the day for scaling/distributed builds on k8s, buildkit has been my personal best-result, and you can even drive k8s-based buildkit instances from the Docker CLI etc using buildx, so it's also the least-painful way for users familiar with the Docker ecosystem to access that wider build ecosystem.

@MattiasMartens

If I understand correctly, you’re saying it’s not merely “hard to implement” but impossible to implement because of a decision about separation of concerns made by BuildKit?

Roughly, yes. I guess maybe it'd be possible if someone managed to write a CNI plugin that could hook up an arbitrary container to the Docker libnetwork-driven overlay networks on the various platforms, and then work out how to bundle that inside the buildx or dockerd binary (which CNI probably makes impossible, as it relies on command-line interfaces and special binary name/paths to function).

However, it's possible (I haven't looked closely in a while) that the in-Docker buildkit instance can't use CNI with its OCI work for some other structural reason, even if such a CNI plugin was available.

Honestly, the simplest code-solution to make docker buildx build --network custom-network work would probably be to recognise when that is used with the default builder, and instead spin up a docker-container builder with the appropriate driver-opt, use it, and then throw it away. It'd be slower that the built-in builder and probably hit a bunch of other niggles, but more-accurately mimic the way things used to work. Same for any other docker build command-line arguments that now require using a docker-container or other external builder. (I suspect something very like this is the likely way forward for ComposeFile support too, but am not certain as I'm mostly speculating on the desired compose workflow, I haven't really used Docker Compose heavily, and even then never with a custom network for container builds.)

Just to be clear, the decision wasn't made by BuildKit. BuildKit is one of the results of those decisions.

"[...] work magically for you"

I meant this in the sense that it used to be a single command-line or Compose-file option, and now you have to do approximately three commands to make it work. As seen by the years of comments on this and related tickets, that's clearly a significant UX difference. In my opinion, anything you can bundle up into a config file that drives a whole ecosystem of tools is "magic" compared to having to maintain shell scripts, and I don't think it's strange to say so.

I would be much happier with the status quo if someone, in the five years of this issue being open, had reported success in binding their custom network to their build environment by leveraging a custom builder. So far everyone who has tried this, including myself, has reported failure. Consider this an open request for a working example.

I've posted two working examples in this ticket just this year: One with docker buildx build run from a shell script and one with Docker compose and docker buildx bake, adapted from other people's posted working examples.

And a quick check shows approximately 0 follow-up comments reporting failure to use those approaches.

@TBBle

Thanks @TBBle for the added context, I feel quite clear about this now.

The inability to leverage DNS for custom networks in the build environment is the final hurdle that might have tripped folks up. I’ve seen some failure reports somewhere, maybe not in this issue but somewhere in the 2 o'clock haze from last night. Presumably they didn’t try your example because that does make the solution quite explicit.

Honestly, the simplest code-solution to make docker buildx build --network custom-network work would probably be to recognise when that is used with the default builder, and instead spin up a docker-container builder with the appropriate driver-opt, use it, and then throw it away. It'd be slower that the built-in builder and probably hit a bunch of other niggles, but more-accurately mimic the way things used to work.

I am in favour of this as a solution, particularly if it can somehow solve the DNS issue. I realize the solution I posted about above (https://github.com/docker/buildx/issues/175#issuecomment-2379790250)—modifying the /etc/hosts file for the build container—would have serious problems (not robust to container’s operating system, invalidates build cache layers each time a container or network is rebuilt)

Which DNS issues do you mean?

There was a long-standing bug with DNS access to the linked services due to BuildKit's resolv.conf mangling needing an opt-out for host-networking mode, but that was fixed in February and was shipped in BuildKit 0.13. That's why I was using image=moby/buildkit:master in my examples, to test that fix and show that hostname-based lookups "just worked" now, compared to the demonstration of the problem in January, and removing the need for the --add-host echo-server:$(docker inspect echo-server | jq '.[0].NetworkSettings.Networks["echo-server"].IPAddress' | tr -d '"\n') workaround posted in 2022.

Edit: Also, if you are looking at my examples, passing --buildkitd-flags "--allow-insecure-entitlement=network.host" to buildx create should not be needed as of Buildx 0.13, it should be automatic when buildx build is passed --network host. But I haven't tested that, Buildx 0.13 came out after I last tried this.

@TBBle

@TBBle I was referring to the hostname-based lookup issue. I see now that is fixed. Thank you.

The command I tried was docker buildx create --name mybuilder --driver-opt network=my-network-name so I was missing --driver docker-container and that’s presumably why it didn’t work, and/or the Buildkit image in use was pre-February.

My usecase is a minimal personal container-based dev/test/stage environment where 'docker—compose' is the only "orchestration" (component-architecture) tool I need there; my git server is local to that "self server" environment, so, its component network is neatly included in many builds (more info on the setup) -- and this issue here is one of the major reasons I'm not using 'docker compose' with buildkit this far. Thank you in advance for addressing it properly.

Also require this option due to build it on restricted environment, despite what environment it will be "run" on.

@dm17

I brought localstack into the mix thinking I'll unit test services in various compose stack without needing to mock either the db or the AWS layers and only if the tests pass, build the final image. IOW the tests run as part of the image building process in a multistage manner where the artifacts from the test stages aren't included in the production target but the production target depends on the test target passing. This eliminates the need for external orchestration and just prevents the build of a bad image altogether.

Only after setting everything up I realised I can't have the build process (the RUN which runs the tests) contact other local services on the same custom network :( . So I'm back to mocking calls to databases (or using sqlite etc) and AWS layers which are very tedious, (quite) error-prone, and not as elegant.

Edit: Not something useful to the thread, but just thought about adding that I completely understand @dm17 's frustration and felt the same anguish as they probably have (from their posts here).

Coming back to this issue. I am trying to convert my CI to use the DOCKER_BUILDKIT=1. I have done stuff according to this step, but I am still getting the error.

I am also setting docker context use.

docker context create remote_dev_server --docker "host=ssh://$DEV_SERVER_USER@$DEV_SERVER_IP"
docker context use remote_dev_server
docker network inspect $NETWORK_NAME
docker buildx create --name dev_buildkit_builder --driver docker-container --driver-opt network=${NETWORK_NAME}
docker buildx use dev_buildkit_builder
docker buildx bake \
      --builder dev_buildkit_builder \
      --progress plain \
      --push
#0 building with "dev_buildkit_builder" instance using docker-container driver
#1 [internal] load local bake definitions
#1 reading docker-compose.yml 764B / 764B done
#1 DONE 0.0s
ERROR: network mode "REDACTED" not supported by buildkit - you can define a custom network for your builder using the network driver-opt in buildx create

This is running in .gitlab-ci.yml

What am I doing wrong?

Navigation Menu

Repository navigation

network mode "custom_network" not supported by buildkit #175

53 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

network mode "custom_network" not supported by buildkit #175

Description

Activity

tonistiigi commented on Oct 24, 2019

tonistiigi commented on Feb 16, 2020

tonistiigi commented on Feb 16, 2020

tonistiigi commented on Feb 16, 2020

alanreyesv commented on May 20, 2021

sudo-bmitch commented on Aug 23, 2021

tonistiigi commented on Sep 1, 2021

53 remaining items

dm17 commented on Apr 9, 2024

MattiasMartens commented on Sep 27, 2024

TBBle commented on Sep 27, 2024

dm17 commented on Sep 27, 2024

MattiasMartens commented on Sep 27, 2024

sudo-bmitch commented on Sep 27, 2024

MattiasMartens commented on Sep 27, 2024

TBBle commented on Sep 27, 2024

MattiasMartens commented on Sep 27, 2024

TBBle commented on Sep 27, 2024

MattiasMartens commented on Sep 27, 2024

costa commented on Mar 16, 2025

gr3m1in commented on May 9, 2025

spandan-sharma commented on Jun 2, 2025

TafkaMax commented on Mar 11, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions