大家好，又见面了，我是你们的朋友全栈君。

背景

最近在分析golang的一个内存泄漏问题。一般来讲，使用golang自带的pprof工具就可以分析内存的使用，协程情况，是否有block等情况。但是我们项目中调用了C库，导致C库的一些东西没法通过pprof来进行监控分析。实际上通过pprof来监控程序的话，内存是稳定的，但是占用Linux的内存是一直增长的，即RES一直增长，实际上程序是有泄漏的。怀疑是使用C库导致，所以通过coredump文件来进行分析。下面主要介绍golang coredump的使用。

分析

主要参考golang官网Debugging Go Code with GDB。
其中：
The GOTRACEBACK variable controls the amount of output generated when a Go program fails due to an unrecovered panic or an unexpected runtime condition. By default, a failure prints a stack trace for the current goroutine, eliding functions internal to the run-time system, and then exits with exit code 2. The failure prints stack traces for all goroutines if there is no current goroutine or the failure is internal to the run-time. GOTRACEBACK=none omits the goroutine stack traces entirely. GOTRACEBACK=single (the default) behaves as described above. GOTRACEBACK=all adds stack traces for all user-created goroutines. GOTRACEBACK=system is like “all” but adds stack frames for run-time functions and shows goroutines created internally by the run-time. GOTRACEBACK=crash is like “system” but crashes in an operating system-specific manner instead of exiting. For example, on Unix systems, the crash raises SIGABRT to trigger a core dump. For historical reasons, the GOTRACEBACK settings 0, 1, and 2 are synonyms for none, all, and system, respectively. The runtime/debug package’s SetTraceback function allows increasing the amount of output at run time, but it cannot reduce the amount below that specified by the environment variable. See https://golang.org/pkg/runtime/debug/#SetTraceback.
GOTRACEBACK变量可以控制程序在崩溃时，stack的输出情况。下面结合具体地程序来分析。

package main

import (
	"time"

	"github.com/astaxie/beego/logs"
)

func main() {
	logs.Info("Start...")
	defer logs.Info("exit.")
	i := 0
	c := make(chan int, 1)
	for {
		go func(i int) {
			mem := make([]int, 100*1024*1024)
			logs.Info("i=%d,mem:%p", i, mem)
			mem[0] = <-c
		}(i)
		i++
		time.Sleep(200 * time.Microsecond)
	}
}

1、首先打开coredump的生成开关: ulimit -c unlimited
2、执行 GOTRACEBACK=crash ./testcoredump，会产生coredump文件

goroutine 279 [running]:
	goroutine running on other thread; stack unavailable
created by main.main
	/opt/gopath/src/test/coredump_test/testcoredump.go:15 +0xdf

goroutine 290 [running]:
	goroutine running on other thread; stack unavailable
created by main.main
	/opt/gopath/src/test/coredump_test/testcoredump.go:15 +0xdf
Aborted (core dumped)

3、gdb可以进行调试，查看程序运行的详细情况

 gdb testcoredump core.15956
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-110.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
...
(gdb) start
Temporary breakpoint 1 at 0x618c50: file /opt/gopath/src/test/coredump_test/testcoredump.go, line 9.
Starting program: /opt/gopath/src/test/coredump_test/testcoredump
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7ffff77f1700 (LWP 15980)]
[New Thread 0x7ffff6ff0700 (LWP 15981)]
[New Thread 0x7ffff5fee700 (LWP 15983)]
[New Thread 0x7ffff67ef700 (LWP 15982)]
[New Thread 0x7ffff57ed700 (LWP 15984)]

Temporary breakpoint 1, main.main () at /opt/gopath/src/test/coredump_test/testcoredump.go:9
9	func main() {
(gdb)

gdb常用命令：

start		//开始调试
n		//一条一条执行
step/s		//执行下一条，如果函数进入函数
backtrace/bt		//查看函数调用栈帧
info/i locals		//查看当前栈帧局部变量
frame/f		//选择栈帧，再查看局部变量
print/p		//打印变量的值
finish		//运行到当前函数返回
set var sum=0		//修改变量值
list/l 行号或函数名		//列出源码
display/undisplay sum		//每次停下显示变量的值/取消跟踪
break/b  行号或函数名		//设置断点
continue/c		//连续运行
info/i breakpoints		//查看已经设置的断点
delete breakpoints 2		//删除某个断点
disable/enable breakpoints 3		//禁用/启用某个断点
break 7 if ok == true		//满足条件才激活断点
run/r		//重新从程序开头连续执行
watch input[7]		//设置观察点
info/i watchpoints		//查看设置的观察点
x/7b input		//打印存储器内容，b--每个字节一组，7--7组
disassemble		//反汇编当前函数或指定函数
 si		// 一条指令一条指令调试 而 s 是一行一行代码
 info registers		// 显示所有寄存器的当前值
x/20 $esp		//查看内存中开始的20个数

总结

程序崩溃，可以通过coredump详细地查看程序调用栈的相关信息，可以更迅速的定位到程序的问题，特别是引起程序崩溃的bug：内存泄漏，一些panic等，当然在写程序时尽量多些log更方便调试。golang自带的pprof在涉及到c库的调用时，会监测不到，这时coredump结合gdb进行调试会比较有用。