Skip to content

feat: return better error if file size is too big to upload #7775

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Jun 5, 2023
Merged

feat: return better error if file size is too big to upload #7775

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
47e3480
feat: return better error if file size is too big to upload
Emyrk Jun 1, 2023
b34e9d8
Fix conditional
Emyrk Jun 1, 2023
0cd070c
Use a limit writer to capture actual tar size
Emyrk Jun 1, 2023
7b2844d
Catch and rewrite error
Emyrk Jun 1, 2023
ec5604e
Add unit test to check edges
Emyrk Jun 1, 2023
3e368a4
t.Parallel unit test
Emyrk Jun 5, 2023
cf35412
fixup! t.Parallel unit test
Emyrk Jun 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
feat: return better error if file size is too big to upload
  • Loading branch information
@Emyrk
Emyrk committed Jun 1, 2023
commit 47e3480d496c8e2e3a84c7551a6f7c0b1074c838
7 changes: 6 additions & 1 deletion provisionersdk/archive.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ func Tar(w io.Writer, directory string, limit int64) error {
)
}

fileTooBigError := xerrors.Errorf("Archive too big. Must be <= %d bytes", limit)
err = filepath.Walk(directory, func(file string, fileInfo os.FileInfo, err error) error {
if err != nil {
return err
Expand Down Expand Up @@ -95,6 +96,10 @@ func Tar(w io.Writer, directory string, limit int64) error {
if !fileInfo.Mode().IsRegular() {
return nil
}
// Before we even open the file, check if it is going to exceed our limit.
if fileInfo.Size()+totalSize >= limit {
return fileTooBigError
}
data, err := os.Open(file)
if err != nil {
return err
Expand All @@ -106,7 +111,7 @@ func Tar(w io.Writer, directory string, limit int64) error {
}
totalSize += wrote
if limit != 0 && totalSize >= limit {
Copy link
Member

@mafredri mafredri Jun 1, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Either the error or this (+ the other) check are wrong? Error says <=, this says >=, who's right?

Another observation is that this doesn't account for tar overhead, AFAICT. So we might end up with an archive that's too large anyway at which point we would hit the original issue again (bad message from API)?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps we can wrap w (wc := writeCounter{w}) before passing it on to tar writer, and count bytes written to it? Then we don't need to track wrote here and just check wc.written.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right about the headers. Honestly I didn't look too deep at the code and just assumed the totalSize was being tracked correctly 🤦.

As for the <= vs >=, I'll change the conditional to > on the check to be consistent with the api.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mafredri good catch. I am using a limit writer now

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Writing a test or two

return xerrors.Errorf("Archive too big. Must be <= %d bytes", limit)
return fileTooBigError
}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I kept this because this is checking the actual bytes written. I can't imagine the fileInfo.Size() being incorrect, as the comment on Size is

// length in bytes for regular files; system-dependent for others

But the others is why I kept this here. Idk how sym links or other edge cases are handled, and then the question of "what does windows do" is not something I care to look into. Doing the check twice is super cheap though, so I can't see this being bad.

return data.Close()
})
Expand Down