3

Introduction

I'm attempting to use the following version of docker on a Linux VM (uname -a returns Linux xen 4.1.17-yocto-standard #1 SMP PREEMPT Thu Jun 2 13:29:47 PDT 2016 x86_64 GNU/Linux), built from the docker_git BitBake recipe.

If I try to run docker version, I get the following output:

Client version: 1.6.2
Client API version: 1.18
Go version (client): go1.3
Git commit (client): 7c8fca2-dirty
OS/Arch (client): linux/amd64

Then the command hangs.

What it should look like

I tried executing docker version on a working docker install (Ubuntu 14.04), and I get the following output:

Client version: 1.6.2
Client API version: 1.18
Go version (client): go1.2.1
Git commit (client): 7c8fca2
OS/Arch (client): linux/amd64
Server version: 1.6.2
Server API version: 1.18
Go version (server): go1.2.1
Git commit (server): 7c8fca2
OS/Arch (server): linux/amd64

So, I'm assuming that there's some sort of error when fetching server info.

Additional Research

I'm not familiar with Go, so this section may be cringe-inducing as I try to figure out what the heck is going on here.

I started looking at this portion of api/client/version.go of the Docker source code:

var versionTemplate = `Client:
 Version:      {{.Client.Version}}
 API version:  {{.Client.APIVersion}}
 Go version:   {{.Client.GoVersion}}
 Git commit:   {{.Client.GitCommit}}
 Built:        {{.Client.BuildTime}}
 OS/Arch:      {{.Client.Os}}/{{.Client.Arch}}{{if .Client.Experimental}}
 Experimental: {{.Client.Experimental}}{{end}}{{if .ServerOK}}
Server:
 Version:      {{.Server.Version}}
 API version:  {{.Server.APIVersion}}
 Go version:   {{.Server.GoVersion}}
 Git commit:   {{.Server.GitCommit}}
 Built:        {{.Server.BuildTime}}
 OS/Arch:      {{.Server.Os}}/{{.Server.Arch}}{{if .Server.Experimental}}
 Experimental: {{.Server.Experimental}}{{end}}{{end}}`

it continues onto this section:

vd := types.VersionResponse{
    Client: &types.Version{
        Version:      dockerversion.Version,
        APIVersion:   cli.client.ClientVersion(),
        GoVersion:    runtime.Version(),
        GitCommit:    dockerversion.GitCommit,
        BuildTime:    dockerversion.BuildTime,
        Os:           runtime.GOOS,
        Arch:         runtime.GOARCH,
        Experimental: utils.ExperimentalBuild(),
    },
}

From engine-api/types/client.go:

// VersionResponse holds version information for the client and the server
type VersionResponse struct {
    Client *Version
    Server *Version
} 

So all that needs to be done at this point is assign something to the Server member (of type *Version). This happens in the section following the vd assignment from above:

serverVersion, err := cli.client.ServerVersion(context.Background())
if err == nil {
    vd.Server = &serverVersion
}

The function definition for ServerVersion is the following from engine-api/client/version.go

// ServerVersion returns information of the docker client and server host.
func (cli *Client) ServerVersion(ctx context.Context) (types.Version, error) {
    resp, err := cli.get(ctx, "/version", nil, nil)
    if err != nil {
        return types.Version{}, err
    }

    var server types.Version
    err = json.NewDecoder(resp.body).Decode(&server)
    ensureReaderClosed(resp)
    return server, err
}

From what I can gather, the above get function call points to client/request.go from Docker's engine API repo

// getWithContext sends an http request to the docker API using the method GET with a specific go context.
func (cli *Client) get(ctx context.Context, path string, query url.Values, headers map[string][]string) (*serverResponse, error) {
    return cli.sendRequest(ctx, "GET", path, query, nil, headers)
}

Where:

  • ctx is context.Background()
  • path is /version
  • no query
  • no headers

And this documentation for sendRequest from vendor/src/google.golang.org/grpc/call.go:

// sendRequest writes out various information of an RPC such as Context and Message.
func sendRequest(ctx context.Context, codec Codec, callHdr *transport.CallHdr, t transport.ClientTransport, args interface{}, opts *transport.Options) (_ *transport.Stream, err error) {
    stream, err := t.NewStream(ctx, callHdr)
    if err != nil {
        return nil, err
    }
    defer func() {
        if err != nil {
            if _, ok := err.(transport.ConnectionError); !ok {
                t.CloseStream(stream, err)
            }
        }
    }()
    // TODO(zhaoq): Support compression.
    outBuf, err := encode(codec, args, compressionNone)
    if err != nil {
        return nil, transport.StreamErrorf(codes.Internal, "grpc: %v", err)
    }
    err = t.Write(stream, outBuf, opts)
    if err != nil {
        return nil, err
    }
    // Sent successfully.
    return stream, nil
}

This has been guesswork for a while and I'm now concerned that I may be looking in the wrong place.

Questions

  • What's causing docker version, docker run hello-world, docker images, docker ps, and docker info to hang and how can it be fixed?
  • Or, is there a more effective way to inspect the cause of this error?
karobar
  • 237
  • 2
  • 14
  • Which OS are you running? Have you tried installing a newer version of docker? – Ken J Jun 10 '16 at 14:11
  • This is a very specific question concerning a very specific project, what just struck me `This package contains the daemon and client. Using docker.io on non-amd64 hosts is not supported at this time.` Are you sure you are running it on AMD processor and not Intel? – Vojtěch Dohnal Jun 13 '16 at 11:47
  • I was running it on a virtual Intel CPU. Also, to Ken's question, the OS is homebrew with linux 4.1 kernel. I've also tried installing 1.11 with similar effects. I'll update the question soon. – karobar Jun 13 '16 at 15:11
  • You should `strace` the process/daemon that listens to the socket at `/var/run/docker.sock`. Your `docker info` command sent a request beginning with "GET /v1.18/info HTTP/1.1\r\nHost:" to the socket and waited forever for a reply, but the server didn't reply. – Deltik Jun 13 '16 at 17:06

1 Answers1

3

Your strace output strongly suggests that the Docker client isn't able to talk to the Docker daemon, which by default creates a socket at /var/run/docker.sock.

The Docker daemon is supposed to be a system service (on systemd, located at /lib/systemd/system/docker.service with the socket configuration at /lib/systemd/system/docker.socket), but it can be started independently using /usr/bin/docker daemon followed by any optional options.

You should strace the daemon rather than the client.

Using strace on the Docker daemon

  1. Get the process ID of your Docker daemon. Either of these commands would store the PID in a variable called $DOCKER_PID.

    • Directly from the socket:

      DOCKER_PID=$(sudo lsof -Ua /var/run/docker.sock | awk '/^docker/ {print $2}' | head -1)
      
    • systemd:

      DOCKER_PID=$(systemctl show -p MainPID docker.service | awk -F'=' '{print $NF}')
      
    • Other:

      DOCKER_PID=$(ps aux | grep 'docker daemon' | grep -v 'grep' | awk '{print $2}' | head -1)
      
  2. Use strace on the Docker daemon, now that you have the PID:

    sudo strace -vvvfts1000 -p $DOCKER_PID
    
  3. In a separate terminal, run a command that normally hangs in the Docker client.

    docker version
    
  4. Watch the strace on the Docker daemon to witness what happens starting at the listening end of the socket.

Interpreting the strace output

Here's what the daemon is supposed to do when you run docker version:

  1. Read what the client sent:

    [pid 14291] 12:34:36 <... read resumed> "GET /v1.22/version HTTP/1.1\r\nHost: \r\nUser-Agent: Docker-Client/1.10.3 (linux)\r\n\r\n", 4096) = 81
    
  2. Collect information about the system:

    [pid 14291] 12:34:36 uname({sysname="Linux", nodename="node51", release="4.4.0-22-generic", version="#40-Ubuntu SMP Thu May 12 22:03:46 UTC 2016", machine="x86_64", domainname="(none)"}) = 0
    
  3. Reply to the client with information about the system:

    [pid 14291] 12:34:36 write(3, "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nServer: Docker/1.10.3 (linux)\r\nDate: Mon, 13 Jun 2016 17:34:36 GMT\r\nContent-Length: 194\r\n\r\n{\"Version\":\"1.10.3\",\"ApiVersion\":\"1.22\",\"GitCommit\":\"20f81dd\",\"GoVersion\":\"go1.6.1\",\"Os\":\"linux\",\"Arch\":\"amd64\",\"KernelVersion\":\"4.4.0-22-generic\",\"BuildTime\":\"Wed, 20 Apr 2016 14:19:16 -0700\"}\n", 334) = 334
    
  4. The client (docker version) then displays the information that the server returned:

    Server:
     Version:      1.10.3
     API version:  1.22
     Go version:   go1.6.1
     Git commit:   20f81dd
     Built:        Wed, 20 Apr 2016 14:19:16 -0700
     OS/Arch:      linux/amd64
    

In your problem, your Docker daemon apparently did not do step #3 because if it had, the client would have seen the reply, but the client did not receive anything.

You should be able to use this information to figure out why the Docker daemon is not replying to requests from the client.

Possible causes

The information that you have provided is not enough to pinpoint the cause of your Docker client's inability to get a reply from the Docker daemon, but here are some tips:

  • Is the Docker daemon running?
  • What happens if you start the Docker daemon in the foreground?: sudo docker daemon
  • Is the Docker daemon listening to the socket at /var/run/docker.sock? sudo lsof -p $DOCKER_PID should show "/var/run/docker.sock type=STREAM" in there somewhere.
  • Are there security policies in place that would block something in the client or daemon? On Linux, SELinux and AppArmor may cause confusion as policies set for them may deny access.
  • In the strace of the daemon, if you don't get an HTTP GET request from the client, that means that the server didn't receive anything from the socket.
  • If you did docker version and see in the strace of the daemon that there was no uname() call, the daemon didn't even try to fetch information about the system.
  • If you see the write() call in the strace of the daemon, it means that the daemon replied, but the client didn't see it.
  • Maybe this is a known issue in the older version of Docker that you're using. Try upgrading.
Deltik
  • 19,353
  • 17
  • 73
  • 114
  • 1
    To me, it was because `docker` was listening on TCP socket only and not on docker.sock. I was going crazy until I understood that (thanks for your complete response which help me a lot). Double-check your `/etc/sysconfig/docker-network` to be sure you have `-H unix:///var/run/docker.sock` – Doomsday Jul 07 '16 at 08:20