Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

dockerd panic: runtime error: slice bounds out of range on CentOS 7 #26384

Closed
tibkiss opened this issue Sep 7, 2016 · 6 comments
Closed

dockerd panic: runtime error: slice bounds out of range on CentOS 7 #26384

tibkiss opened this issue Sep 7, 2016 · 6 comments
Labels
area/runtime kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. version/1.12
Milestone
1.12.2

Comments

@tibkiss
Copy link

tibkiss commented Sep 7, 2016 •
edited
Loading

Output of docker version:

Client:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 6
 Running: 4
 Paused: 0
 Stopped: 2
Images: 121
Server Version: 1.12.1
Storage Driver: devicemapper
 Pool Name: docker-253:0-509713-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 6.043 GB
 Data Space Total: 107.4 GB
 Data Space Available: 45.74 GB
 Metadata Space Used: 9.527 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.138 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-327.28.3.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 11.46 GiB
Name: meru.localdomain
ID: IHRD:QPSF:PNIE:MIMP:TEQA:LNYQ:MF2P:MYHO:SGSW:IBJF:4NB7:RUGN
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.):
physical machine

Steps to reproduce the issue:

  1. Start any python application using {{pypy}} Docker image
  2. Note: The error is intermittent.

Describe the results you received:
After starting the application in pypy container the dockerd panics with the following error:

Sep 07 08:10:32 meru.localdomain dockerd[28751]: panic: runtime error: slice bounds out of range
Sep 07 08:10:32 meru.localdomain dockerd[28751]: goroutine 46310 [running]:
Sep 07 08:10:32 meru.localdomain dockerd[28751]: panic(0x1a7edc0, 0xc82000a040)
Sep 07 08:10:32 meru.localdomain dockerd[28751]: /usr/local/go/src/runtime/panic.go:481 +0x3e6
Sep 07 08:10:32 meru.localdomain dockerd[28751]: bytes.(*Buffer).grow(0xc8232bc600, 0x2eb, 0xc823376de0)
Sep 07 08:10:32 meru.localdomain dockerd[28751]: /usr/local/go/src/bytes/buffer.go:112 +0x141
Sep 07 08:10:32 meru.localdomain dockerd[28751]: bytes.(*Buffer).Write(0xc8232bc600, 0xc820f56000, 0x2eb, 0x8000, 0x12b, 0x0, 0x0)
Sep 07 08:10:32 meru.localdomain dockerd[28751]: /usr/local/go/src/bytes/buffer.go:134 +0x4b
Sep 07 08:10:32 meru.localdomain dockerd[28751]: github.com/docker/docker/daemon.(*limitedBuffer).Write(0xc8232bc600, 0xc820f56000, 0x2eb, 0x8000, 0x2eb, 0x0, 0x0)
Sep 07 08:10:32 meru.localdomain dockerd[28751]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/daemon/health.go:283 +0xae
Sep 07 08:10:32 meru.localdomain dockerd[28751]: io.copyBuffer(0x7f3cad02ce18, 0xc8232bc600, 0x7f3cacfe8ff0, 0xc82327e380, 0xc820f56000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
Sep 07 08:10:32 meru.localdomain dockerd[28751]: /usr/local/go/src/io/io.go:382 +0x2c9
Sep 07 08:10:32 meru.localdomain dockerd[28751]: io.Copy(0x7f3cad02ce18, 0xc8232bc600, 0x7f3cacfe8ff0, 0xc82327e380, 0xc82327e380, 0x0, 0x0)
Sep 07 08:10:32 meru.localdomain dockerd[28751]: /usr/local/go/src/io/io.go:350 +0x64
Sep 07 08:10:32 meru.localdomain dockerd[28751]: github.com/docker/docker/container.AttachStreams.func2(0x1d64ec0, 0x6, 0x7f3cad02ce18, 0xc8232bc600, 0x7f3cacfe8fc0, 0xc82327e380)
Sep 07 08:10:32 meru.localdomain dockerd[28751]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/container/container.go:433 +0x1cc
Sep 07 08:10:32 meru.localdomain dockerd[28751]: created by github.com/docker/docker/container.AttachStreams
Sep 07 08:10:32 meru.localdomain dockerd[28751]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/container/container.go:450 +0x37e

Describe the results you expected:
dockerd should not panic.

Additional information you deem important (e.g. issue happens only occasionally):
Intermittent problem.
@cpuguy83 cpuguy83 added this to the 1.12.2 milestone Sep 7, 2016
@cpuguy83 cpuguy83 added kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. area/runtime labels Sep 7, 2016
@cpuguy83 cpuguy83 mentioned this issue Sep 8, 2016
Closed
@fabiop
Copy link

fabiop commented Sep 8, 2016

Hello, I'm having a similar issue:

Running a 1.12.1 swarm cluster with 5 managers and 2 nodes, hosting one service.

Containers are normally deployed all across the cluster, between managers and nodes.

After a random number of minutes Docker on one or both of the nodes is doing down. The rest of the cluster (meaning all the managers) is fine, the containers that were hosted on the dead node(s) are being rescheduled on the managers, so the total number of containers in the cluster remains the same (after the moved ones become available).

'docker node ls' on the managers indicates the node(s) state as "Down".

root@xxxxxxx-public-manager-i-77acb8fb:~# docker node ls
ID                           HOSTNAME                           STATUS  AVAILABILITY  MANAGER STATUS
3s0u1btbzvh9qcoesxiwforff    xxxxxxx-public-manager-i-f2a5007f  Ready   Active        Reachable
7ycxht9cas1lhqai20deg70r5    xxxxxxx-public-worker-i-15862298   Down    Active
9c3eyuqde1gxz1ubfwl4pogx6    xxxxxxx-public-manager-i-47c86dca  Ready   Active        Reachable
b39d59z7g6m0dz00fep2jxbwx    xxxxxxx-public-worker-i-0d5cbd3c   Down    Active
bftd5su9tn5stcyxv2z7q67zr    xxxxxxx-public-manager-i-0ba1403a  Ready   Active        Reachable
c7sz4lhvauybbihhd3bg8yfzn *  xxxxxxx-public-manager-i-77acb8fb  Ready   Active        Leader
ex9c362g7q9ed5vb0hnf1oqht    xxxxxxx-public-manager-i-8daf4ebc  Ready   Active        Reachable
root@xxxxxxx-public-manager-i-77acb8fb:~#

ssh into the nodes and doing a 'docker ps' will hang for a couple of seconds, then that will awaken Docker again and output the docker ps header with no containers.

The outcome above has been replicated a number of times, on brand new deployed clusters with different AWS instances (same config scripted via terraform)

The context is:
AWS, 1 ASG for managers, 1 ASG for nodes, Ubuntu 16.04.1 LTS on all instances.

Log output and debug info:

This is the log on one of the nodes after going down:

root@xxxxxxx-public-worker-i-d3cb6e5e:~# journalctl -u docker
Sep 08 10:21:46 xxxxxxx-public-worker-i-0d5cbd3c dockerd[5511]: panic: runtime error: slice bounds out of range
Sep 08 10:21:46 xxxxxxx-public-worker-i-0d5cbd3c dockerd[5511]: goroutine 64552 [running]:
Sep 08 10:21:46 xxxxxxx-public-worker-i-0d5cbd3c dockerd[5511]: panic(0x1a84380, 0xc82000e050)
Sep 08 10:21:46 xxxxxxx-public-worker-i-0d5cbd3c dockerd[5511]:         /usr/local/go/src/runtime/panic.go:481 +0x3e6
Sep 08 10:21:46 xxxxxxx-public-worker-i-0d5cbd3c dockerd[5511]: bytes.(*Buffer).grow(0xc821601980, 0x1d9, 0xc821e3bcb0)
Sep 08 10:21:46 xxxxxxx-public-worker-i-0d5cbd3c dockerd[5511]:         /usr/local/go/src/bytes/buffer.go:112 +0x141
Sep 08 10:21:46 xxxxxxx-public-worker-i-0d5cbd3c dockerd[5511]: bytes.(*Buffer).Write(0xc821601980, 0xc8211ec000, 0x1d9, 0x8000, 0x19, 0x0, 0x0)
Sep 08 10:21:46 xxxxxxx-public-worker-i-0d5cbd3c dockerd[5511]:         /usr/local/go/src/bytes/buffer.go:134 +0x4b
Sep 08 10:21:46 xxxxxxx-public-worker-i-0d5cbd3c dockerd[5511]: github.com/docker/docker/daemon.(*limitedBuffer).Write(0xc821601980, 0xc8211ec000, 0x1d9, 0x8000, 0x1d9, 0x0, 0x0)
Sep 08 10:21:46 xxxxxxx-public-worker-i-0d5cbd3c dockerd[5511]:         /usr/src/docker/.gopath/src/github.com/docker/docker/daemon/health.go:283 +0xae
Sep 08 10:21:46 xxxxxxx-public-worker-i-0d5cbd3c dockerd[5511]: io.copyBuffer(0x7f4c2dd09270, 0xc821601980, 0x7f4c2dd08d68, 0xc820def540, 0xc8211ec000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
Sep 08 10:21:46 xxxxxxx-public-worker-i-0d5cbd3c dockerd[5511]:         /usr/local/go/src/io/io.go:382 +0x2c9
Sep 08 10:21:46 xxxxxxx-public-worker-i-0d5cbd3c dockerd[5511]: io.Copy(0x7f4c2dd09270, 0xc821601980, 0x7f4c2dd08d68, 0xc820def540, 0xc820def540, 0x0, 0x0)
Sep 08 10:21:46 xxxxxxx-public-worker-i-0d5cbd3c dockerd[5511]:         /usr/local/go/src/io/io.go:350 +0x64
Sep 08 10:21:46 xxxxxxx-public-worker-i-0d5cbd3c dockerd[5511]: github.com/docker/docker/container.AttachStreams.func2(0x1d6a480, 0x6, 0x7f4c2dd09270, 0xc821601980, 0x7f4c2dd08d38, 0xc820def540)
Sep 08 10:21:46 xxxxxxx-public-worker-i-0d5cbd3c dockerd[5511]:         /usr/src/docker/.gopath/src/github.com/docker/docker/container/container.go:433 +0x1cc
Sep 08 10:21:46 xxxxxxx-public-worker-i-0d5cbd3c dockerd[5511]: created by github.com/docker/docker/container.AttachStreams
Sep 08 10:21:46 xxxxxxx-public-worker-i-0d5cbd3c dockerd[5511]:         /usr/src/docker/.gopath/src/github.com/docker/docker/container/container.go:450 +0x37e
Sep 08 10:21:46 xxxxxxx-public-worker-i-0d5cbd3c systemd[1]: docker.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Sep 08 10:21:46 xxxxxxx-public-worker-i-0d5cbd3c systemd[1]: docker.service: Unit entered failed state.
Sep 08 10:21:46 xxxxxxx-public-worker-i-0d5cbd3c systemd[1]: docker.service: Failed with result 'exit-code'.

Docker version:

root@xxxxxxx-public-worker-i-0d5cbd3c:~# docker version
Client:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 05:33:38 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 05:33:38 2016
 OS/Arch:      linux/amd64
root@xxxxxxx-public-worker-i-0d5cbd3c:~#

Docker info:

root@xxxxxxx-public-worker-i-0d5cbd3c:~# docker info
Containers: 9
 Running: 0
 Paused: 0
 Stopped: 9
Images: 1
Server Version: 1.12.1
Storage Driver: devicemapper
 Pool Name: docker-202:1-183116-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 138 MB
 Data Space Total: 107.4 GB
 Data Space Available: 7.866 GB
 Metadata Space Used: 962.6 kB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.147 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.110 (2015-10-30)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null bridge overlay host
Swarm: active
 NodeID: b39d59z7g6m0dz00fep2jxbwx
 Is Manager: false
 Node Address: <redacted>
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-36-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.303 GiB
Name: xxxxxxx-public-worker-i-0d5cbd3c
ID: OQVN:WWEN:54EJ:CMHS:DSHA:CW6E:IS7J:LD5H:HLHF:HSWU:H3BH:EFN7
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8
root@xxxxxxx-public-worker-i-0d5cbd3c:~#

Please let me know if any other debug info are needed and thanks for helping!

@stephendrake stephendrake mentioned this issue Sep 12, 2016
Merged
@stephendrake
Copy link
Contributor

This is caused by a bug in stdout / stderr processing in Docker 1.12's health checks.

Until the next release is available, a workaround to prevent dockerd from panicking is to make sure that stdout and stderr aren't written to at the same time by the command(s) used in a HEALTHCHECK.

e.g.

HEALTHCHECK CMD curl -f 'http://localhost:1234/MyHealthTest' > /dev/null 2>&1 || exit 1

Alternatively, if you would still like to see healthckeck output in docker inspect

HEALTHCHECK CMD curl -f 'http://localhost:1234/MyHealthTest' 2>&1 || exit 1

@fabiop
Copy link

fabiop commented Sep 14, 2016 •
edited
Loading

Great, will give this a try, thanks @drakenator

Update: this workaround is working well on my setup, thanks

@stephendrake stephendrake mentioned this issue Sep 15, 2016
Merged
@thaJeztah
Copy link
Member

Looks like this is resolved in the 1.12.x branch, and will be fixed on master through #26596

@tibkiss
Copy link
Author

tibkiss commented Sep 15, 2016

Thanks for the quick fix & workaround @drakenator . I'm testing it since yesterday: so far so good!

@mlaventure mlaventure mentioned this issue Sep 23, 2016
Closed
@BrickXu
Copy link

BrickXu commented Apr 27, 2017

Thanks to all of you, I meet this bug just now and this issue helps me so much.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/runtime kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. version/1.12
Projects
None yet
Milestone
1.12.2
Development

No branches or pull requests
7 participants
@tibkiss @stephendrake @fabiop @cpuguy83 @thaJeztah @BrickXu @GordonTheTurtle