Docker containers cannot connect to internet in Fedora 32?

Question

The Problem

I have recently created a docker container that has to pull a public github repository, however, it isn't able to resolve the host github.com. In fact, it isn't able to even execute ping command.

NOTE: all network operations here are inside the container after building, when the container runs

ping www.google.com gives cannot resolve host

It cannot connect to the internet at all.

After looking on the internet and Docker Documentation

Solutions Tried

1) Enabling IP Forwarding as given in Docker Documentation

I tried enabling IPForwarding but to no avail.

Contents of /usr/lib/systemd/network/80-container-host0.network after enabling IPForwarding

...
[Network]
DHCP=yes
LinkLocalAddressing=yes
LLDP=yes
EmitLLDP=customer-bridge
IPForward=true // this line was changed
[DHCP]
UseTimezone=yes

2) Recreating Docker Bridge as given in Stack Overflow

pkill docker
iptables -t nat -F
ifconfig docker0 down
brctl delbr docker0
sudo service restart docker

This was originally effective in this issue

There is a solution for ubuntu, but my network configuration doesn't have that problematic line

my sudo pico /etc/NetworkManager/NetworkManager.conf is just filled with comments

[main]
#plugins=ifcfg-rh


[logging]
#level=TRACE
#domains=ALL

3) Installing IP-Tables service StackOverflow

sudo yum install iptables-services
sudo service docker restart

4) Overriding DNS configuration in docker-compose-StackOverflow

version: "3.3"
services:
    airflow:
        build: 
            context: ./airflow
            dockerfile: Dockerfile
        ports: 
            - 8080:8080
        environment: 
            GITHUB_DAG_REPO: https://github.com/siddharths067/HelloAirflow.git
        dns:
            - 8.8.8.8
            - 8.8.4.4

The Docker Network Inspect of the Image's Network

I don't know if this would be of any use

docker network inspect airflowsetup_default 
[
    {
        "Name": "airflowsetup_default",
        "Id": "141a518c1440e603f75774c54f42de33e9173e3f062a0a0bc772db13a7f1ef5d",
        "Created": "2020-08-30T14:42:30.951975699+05:30",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "default",
            "com.docker.compose.project": "airflowsetup",
            "com.docker.compose.version": "1.25.4"
        }
    }
]

Output

airflow_1  | GITHUB DAG REPO IS
airflow_1  | https://github.com/siddharths067/HelloAirflow.git
airflow_1  | Cloning into 'HelloAirflow'...
airflow_1  | fatal: unable to access 'https://github.com/siddharths067/HelloAirflow.git/': Could not resolve host: github.com

check my answer for the most likely root cause of your problem and a possible solution. — Konrad Botor, Sep 06 '20 at 13:57

Konrad Botor · Accepted Answer · 2020-09-07T12:56:29.787

Event though the OP found a workaround for his problem and closed this issue, he did not find the root issue. The fact that Docker's default bridged network bridge is connected to the internet and airflowsetup_default is not suggests that something is wrong with Docker networking setup.

I did some research and it turns Fedora 32 decided it doesn't really care if Docker works on it or not.

It's not even possible to install Docker in a way described in the documentation and if you install the package provided by Fedora it still doesn't work correctly - more information on that issue can be found here, here and here.

The main issue is that there is no internet connectivity inside the containers if they are connected to any custom bridged network - whether it was created using docker network create or by docker-compose.

The reason why is simple - Docker assumes that the firewall used by the OS is iptables, but Fedora 32 uses firewalld by default. That means Docker cannot configure the firewall manually - it has to be configured by hand.

For reference purposes I will first describe how to set up Docker on a clean Fedora 32 installation.

First run the following commands:

sudo grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0"
sudo groupadd docker
sudo usermod -aG docker $USER

This will configure cgroups to be compatible with Docker daemon and allow your user to use Docker CLI without sudo.

Next restart your system to apply changes and run:

sudo dnf install -y moby-engine docker-compose nano
sudo systemctl enable docker
sudo systemctl start docker

to install and enable Docker.

Now go to the directory that contains your docker-compose.yml and run docker-compose up -d. Yuo should see docker-compose creating a network for you and then creating the container. If your container requires internet connection on startup (like the OP's) it will fail to start.

Now run sudo iptables-save | grep DOCKER and you should see something like:

:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o br-b56fa303f315 -j DOCKER
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-b56fa303f315 ! -o br-b56fa303f315 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-b56fa303f315 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A DOCKER -i docker0 -j RETURN
-A DOCKER -i br-b56fa303f315 -j RETURN

docker0 is Docker's default bridged network, while br-b56fa303f315 is the new network created by docker-compose (yours may have a different name). If your OS were using iptables everything would work as expected, but it does not, so we need to check the configuration of docker0 in firewalld.

Run firewall-cmd --get-active-zones and you'll get something similar to:

docker
  interfaces: docker0
public
  interfaces: eth0 eth1

You can see that bridge network is in docker zone, but the new network is not. In fact it's not listed at all, which means it's in the default zone. You can check what that is by running firewall-cmd --get-default-zone. On a clean install of Fedora 32 it's public.

So run (remember to replace br-b56fa303f315 with your interface name):

sudo firewall-cmd --zone=docker --add-interface=br-b56fa303f315

run docker-compose up -d if your service failed to start previously and voila - your container has network connectivity.

Unfortunately if your restart your system it'll lose that connectivity again.

You can prevent that by using:

sudo firewall-cmd --permanent --zone=docker --add-interface=br-b56fa303f315
sudo firewall-cmd --reload

However if you create any new networks or recreate existing ones (for example by running docker-compose down and then docker-compose up -d again) you'll have to repeat the process.

So what is the solution to this problem?

First, write down all network interfaces that are currently attached to the default zone - in this example eth0 and eth1.

Then run the following (replacing public with your default zone's name)

sudo firewall-cmd --set-default-zone=docker
sudo firewall-cmd --permanent --zone=public --add-interface=eth0
sudo firewall-cmd --permanent --zone=public --add-interface=eth1
sudo firewall-cmd --reload

Now the interfaces that were previously in your default zone should be there again, but all the new interfaces (and therefore all the new Docker networks) will be automatically added to docker zone, which will give them full network connectivity.

Holy Shit, no wonder why I wasn't able to figure it out. This is too complex for me to figure it out on my own. Can I install iptables in fedora? would that work? — Siddharth Singh, Sep 07 '20 at 12:09
It’s not that simple. I didn’t want to confuse the issue even more in my answer, but in Linux firewall was always a part of the kernel. First it was `ipfirewall`, then `ipchains`, then `iptables` and now `nftables`. Other “firewalls”, including `firewalld`, are merely frontends that make managing them easier. You can read the reasons why Docker used to work with `firewalld` in Fedora 31, but not 32 here: https://fedoraproject.org/wiki/Changes/firewalld_default_to_nftables — Konrad Botor, Sep 07 '20 at 12:51
I think setting the default zone to `docker` is the best solution, but you may also try forcing `firewalld` to use `iptables` as described here: https://dev.to/ozorest/fedora-32-how-to-solve-docker-internal-network-issue-22me — Konrad Botor, Sep 07 '20 at 12:52
When I fetch active zones I don't even get `public` interfaces https://pastebin.pl/view/7cf674b7 — Siddharth Singh, Sep 09 '20 at 14:06
The output of that command depends on your current system configuration. It's possible that you have no interfaces in `public` zone. Check what zones you do have: `firewall-cmd --get-zones`. If there's one named `docker`, you can still do `sudo firewall-cmd --set-default-zone=docker` to fix Docker networks. Otherwise `sudo firewall-cmd --set-default-zone=trusted` should work, but will make your system less secure. — Konrad Botor, Sep 09 '20 at 14:33

score 2 · Answer 2 · answered Dec 08 '20 at 12:56

I followed an alternative solution from the Fedora Magazine:

Whitelist docker in firewall

To allow Docker to have network access, two commands are needed.
sudo firewall-cmd --permanent --zone=trusted --add-interface=docker0
sudo firewall-cmd --permanent --zone=FedoraWorkstation
--add-masquerade
The first command will add the Docker-interface to the trusted environment which allows Docker to make remote connections. The second command will allow docker to make local connections. This is particularly useful when multiple Docker containers are in as a development environment.

Then you should run this command to validate the changes

sudo firewall-cmd --reload

Normally, you don't have to reboot your computer and your next containers are going to be connected.

In case, it does not work, you could try to:

First, restart a new container
Second, restart docker: sudo systemctl restart docker
Finally, reboot your computer

That did the trick here (Fedora 33). – vonbrand Apr 06 '21 at 18:04 — vonbrand, Apr 06 '21 at 18:04

score 0 · Answer 3 · answered Aug 30 '20 at 10:31

0

Adding network-mode: bridge to my docker-compose file seems to work. But I feel this might not be the proper way to do this. I shouldn't have to over-ride anything to allow my containers to connect.

version: "3.3"
services:
    airflow:
        build: 
            context: ./airflow
            dockerfile: Dockerfile
        ports: 
            - 8080:8080
        environment: 
            GITHUB_DAG_REPO: https://github.com/siddharths067/HelloAirflow.git
        # dns:
        #     - 8.8.8.8
        #     - 8.8.4.4
        network_mode: bridge

answered Aug 30 '20 at 10:31

Siddharth Singh

43
1
5

This is strange - as far as I know setting `network_mode` to `bridge` forces the container to connect to default Docker bridged network rather than to the project-specific bridged network created by default by docker-compose. Did you use top-level `networks` key to customize that project-specific network? Maybe that is the problem. – Konrad Botor Sep 05 '20 at 22:16
I will be honest, I don't understand what you said. I just searched over internet for docker files and used this setting :( – Siddharth Singh Sep 06 '20 at 04:41
I meant that when you specify `network_mode: bridge`, you are connecting your container to a network called `bridge` - which was created during Docker installation - rather than to network `airflowsetup_default` - which is created by docker-compose. – Konrad Botor Sep 06 '20 at 11:49