-
Notifications
You must be signed in to change notification settings - Fork 787
memory leak with clone #315
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Any progress on this issue .... I also found that clone depths set to 1 also pull down the whole thing, unlike
it has a different effect 这个问题有啥进展么....我还发现 clone depth 设置为 1 的也会整个拉下来, 和 git clone --depth
|
Here is for info
|
Hi, any updates on this? I may be experiencing something similar.. @axetroy, did you happen to find a way around that? |
i am having same OOM issue |
Since this were never solved, I had to revert to cloning via executing the git command itself from the app.. Very unfortunate. |
To help us keep things tidy and focus on the active tasks, we've introduced a stale bot to spot issues/PRs that haven't had any activity in a while. This particular issue hasn't had any updates or activity in the past 90 days, so it's been labeled as 'stale'. If it remains inactive for the next 30 days, it'll be automatically closed. We understand everyone's busy, but if this issue is still important to you, please feel free to add a comment or make an update to keep it active. Thanks for your understanding and cooperation! |
The ultimate solution to this issue is to use the git command. Each clone is an independent process, and the process exits after completion. There is no need to worry about memory leaks, and the git command is much faster than go-git. // 拉取代码
func (p *Puller) Pull(ctx context.Context) error {
p.logger.Printf("Pulling %s\n", p.options.Repo)
p.logger.Printf("拉取项目到目录 %s\n", p.options.Dir)
switch {
// 按 Hash 克隆
case p.options.Hash != nil && *p.options.Hash != "":
if err := util.EnsureDir(p.options.Dir); err != nil {
return errors.WithStack(err)
}
if err := shell.Run(ctx, "git init", p.options.Writer, p.options.Dir); err != nil {
return errors.WithStack(err)
}
if err := shell.Run(ctx, fmt.Sprintf("git fetch --no-tags --recurse-submodules --depth 1 --no-write-fetch-head %s %s", p.options.Repo, *p.options.Hash), p.options.Writer, p.options.Dir); err != nil {
return errors.WithStack(err)
}
if err := shell.Run(ctx, fmt.Sprintf("git checkout %s", *p.options.Hash), p.options.Writer, p.options.Dir); err != nil {
return errors.WithStack(err)
}
// 按 Tag 克隆
case p.options.Tag != "":
fallthrough
// 按分支克隆
case p.options.Branch != "":
var target string
if p.options.Tag != "" {
target = p.options.Tag
} else {
target = p.options.Branch
}
parentFolder := filepath.Dir(p.options.Dir)
folderName := filepath.Base(p.options.Dir)
if err := os.RemoveAll(p.options.Dir); err != nil {
return errors.WithStack(err)
}
if err := shell.Run(ctx, fmt.Sprintf("git clone --single-branch --depth 1 --no-tags --progress --branch %s %s %s", target, p.options.Repo, folderName), p.options.Writer, parentFolder); err != nil {
return errors.WithStack(err)
}
default:
parentFolder := filepath.Dir(p.options.Dir)
folderName := filepath.Base(p.options.Dir)
if err := util.EnsureDir(parentFolder); err != nil {
return errors.WithStack(err)
}
if err := shell.Run(ctx, fmt.Sprintf("git clone %s %s", p.options.Repo, folderName), p.options.Writer, parentFolder); err != nil {
return errors.WithStack(err)
}
}
if p.options.Recurse {
if err := shell.Run(ctx, "git submodule update --init --recursive --single-branch --depth 1 --checkout --force --progress", p.options.Writer, p.options.Dir); err != nil {
return errors.WithStack(err)
}
}
// 更新子模块
if p.options.Submodule != nil {
for submodule, config := range *p.options.Submodule {
if !config.Latest {
continue
}
if err := shell.Run(ctx, fmt.Sprintf("git submodule update --remote --recursive --depth 1 --progress %s", submodule), p.options.Writer, p.options.Dir); err != nil {
return errors.WithStack(err)
}
}
}
return nil
} |
Yup.. I unfortunately had to abandon using go-git for the most part.. If I have to bundle git executable in my container anyway just to clone, I might as well do the rest with it too, mostly faster and takes less ram. I wish this wasn't the case, but it is what it is 🤷♂️ |
The current implementation of go-git has a high memory cost. Since our needs are not that complex, we can use the default git CLI. Reference: go-git/go-git#315
The current implementation of go-git has a high memory cost. Since our needs are not that complex, we can use the default git CLI. Reference: go-git/go-git#315
When I cloned a repo about 1G in size, it consumed my 1G of memory and generated a lot of stacks
The text was updated successfully, but these errors were encountered: