-
Notifications
You must be signed in to change notification settings - Fork 1.8k
[BUG] Extremely poor performance with slow readFile and slow writeFile on very small files #1842
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
We had similar problem when the input xlsx had data validation set to the whole column (with few first rows excluded). When limiting it to the first few hundred rows (something like rows 3-500), the issue disappeared. |
For data with around 3000 rows, 13 rows, exceljs could generate Excel fast. |
Same to me, Write file 700kb 4K+ row used 2s on m1 and 9s on server |
I was running into a problem like this on a 49kb file where it wouldn't finish loading in even after literally days. I did a deep dive into this and this is what I found. When loading in a file with data validations applied to ranges of cells, it enumerates the address of each individual cell which the validation applies to and stores that (see https://github.com/exceljs/exceljs/blob/master/lib/xlsx/xform/sheet/data-validations-xform.js#L218). This goes relatively quickly (a little less than a millisecond per address on my machine) until it hits 8,388,607 total cell addresses across all validations, at which point it immediately slows WAY down (~3 seconds per address on my machine). As it happened my small spreadsheet had a lot of data validations applied to large ranges (well past what actually existed in the spreadsheet yet). Here's the odd thing though, |
I just ran into the same thing with a weird (but non-malicious) spreadsheet that got uploaded to a service I'm working on. Somehow it ended up containing this crazy data validation: <dataValidation type="list" allowBlank="1" showInputMessage="1" showErrorMessage="1" sqref="F1583:F1048576 G1583:I1048576 Q1583:Q1048576 U1583:AB1048576 S1583:S1048576 L1583:L1048576 AD1583:AE1048576"><formula1>#REF!</formula1></dataValidation> ... which then makes exceljs loop over 17,798,898 cells and set a hash key for each(!) I came up with this super hacky workaround to turn require('exceljs/lib/doc/range').prototype.forEachAddress = function () {}; It would be nice to avoid that. In my application I don't really need these data validations to be applied, so it would be great if they could be turned off. Alternatively we could use the existing |
any updates on this? |
I've just run into this exact issue too. My 2c is that it seems Google Sheets cuts off these data ranges with 1048576 rows (the maximum that Excel supports btw) at 1000 rows. My "fix" for this is to limit the aforementioned forEachAddress(cb) {
for (let col = this.left; col <= this.right; col++) {
for (let row = this.top; row <= Math.min(this.bottom, 1000); row++) {
cb(colCache.encodeAddress(row, col), row, col);
}
}
} Maybe a contributor that knows more might be able to limit it to the actual number or rows that have data in them if that exists? |
🐛 Bug Report
Many files are extremely slow to read and write, despite being very small, eg less than 100kb. For example a file with 2 sheets, columns A - L, 200 rows.
This occurs when reading/writing on the API side, ie using nodejs
Lib version: 4.3.0. Node v10 on a Docker image.
Was also occurring on exceljs 3.x
Steps To Reproduce
Logs output:
Also, occasionally it crashes with an out of memory error when writing the file, which indicates that the slow performance is due to excessive memory use. This is a 67kb file. It should not cause an out of memory error.
Hopefully the stacktrace helps to find where this is occurring.
I have tried the
patchDefinedNames
fix suggested in other issues however it does not work becauseDefinedNames.prototype
is undefined:Log output from the fix is:
This error occurs because
DefinedNames.prototype
is undefined. So this suggested fix does not work as at v4.3.0Note that there are many other reported issues regarding read/write performance, and it does not look like it is being fixed. Instead it looks like this is a serious and ongoing issue.
#535,
#1494,
#1609,
#1017,
#1597,
#847,
#742,
#566,
#535,
#527,
#282,
#125,
The expected behaviour:
File should be read in sub 1 second time.
File should write in 1 or 2 seconds time. Obviously file operations have some overhead.
If this not fixable then sadly we will have to stop using exceljs
The text was updated successfully, but these errors were encountered: