It would probably be best to build sql.js with ALLOW_MEMORY_GROWTH (option 2). Otherwise you can edit TOTAL_MEMORY inside the build.

Do you have any advice on how to do that? Rebuild that is. I tried setting the TOTAL_MEMORY but it's still ocurring

Look for oa=r.TOTAL_MEMORY||16777216, that is where it checks if the input r object has TOTAL_MEMORY on it, and if so uses it, otherwise uses a default 16MB. You can just change that to oa=X for some X in bytes.

I had set oa=r.TOTAL_MEMORY||536870912 for 512MB, but you're saying the TOTAL_MEMORY var is possibly being set elsewhere, which makes my value unreachable right?

Yes.

Alright, thank you. Cmake wasn't playing ball so I was unable to compile with the ALLOW_MEMORY_GROWTH flag. Hopefully this sorts it out.

Just a follow up on this (and thank you for your help BTW) My users are trying to load sql lite files that are ranging from 200MB to 4GB in size. Somewhere around the 2GB mark they're hitting "out of memory" and "invalid array buffer length" errors

My exception reporting is showing this line for the invalid array buffer length issue

var t = new Uint8Array(o.result);
      s.results.db = new SQL.Database(t)

Do you think recompiling with ALLOW_MEMORY_GROWTH set would help? It seems the TOTAL_MEMORY config hasn't completely solved the problem

JS engines probably won't let you allocate a 2GB typed array. In practice, in browsers even 512MB is unreliable due to memory fragmentation.

Hmm, I was worried about that. So, would you say it's not going to be possible?

Well 2GB should be possible in theory - the spec does not preclude it - but I just don't think you'll see it work in practice. Might be worth seeing if there are already bugs filed on this on browsers, likely there are. Also possible it works in some browsers but not others.

Personally, I would guess that 2GB is unlikely to work, for now.

Note that over 2GB would require a change to emscripten, as our pointers are 32-bit signed.

It appears some people have been able to successfully load a 2GB file, although its rare. Most people say it crashes. Others say Safari seems to be more stable than Chrome and FireFox for loading the bigger files. I'm running out of ideas.
Am I right in assuming that when loading the sql file, being a typed array it loads the entire thing into memory? So a 2GB file will require a 2GB heap?
By the way, the tool is here: https://www.lightroomdashboard.com , made possible by your amazing code, I was really hoping this wasn't going to road block the project, but i'm not so sure now.

I don't know about that question. The first issue is whether sqlite loads the entire file into memory. If it does, then yes - a full 2GB would be needed. If not, and sqlite instead reads slice by slice, then this might be done in emscripten, with some work. By default, when we preload a file, we do load it all into memory, and I think we use a typed array. But perhaps making us use a JS array instead would avoid that - if the browser can load a 2GB JS array, but not a 2GB typed array (which I don't know). The relevant code is in src/library_memfs.js (MEMFS is the default filesystem).

Another option might be to write a MEMFS alternative, that loads files in separate chunks. That still requires 2GB of memory, but not contiguous. I don't know how much that would help.

Finally, avoiding 2GB entirely would require synchronous filesystem access - to load a chunk as it is needed, then free it. But the web doesn't have sync filesystem access. Web workers, however, do have synchronous Blob reading, so perhaps a solution could be done if the app runs in a worker.

Overall, none of those options is easy to do, and all seem to have some risk. I'm afraid that loading 2GB+ files is just something browsers don't do well yet. Game engines and others have been complaining about this, so progress is being made, but it takes a while. As more browsers become 64-bit, that will help, for example.

Perhaps you can work around this in some other way? Split up big sqlite databases?

Thanks for the detailed explanation. Yes it sounds like none of those are easy to achieve. Unfortunately I can't control the files being loaded, these are from users machines, I just never expected them to be loading such large files.
Reworking emscripten is a little over my head I think, i'm not sure I would know where to start there.

Regarding what we spoke about earlier, changing the TOTAL_MEMORY value, can you help me understand this a little better? What exactly is this setting a limit for? I set this to 256MB, does that mean the ceiling limit for loading these sql files into memory is 256MB? So, allocating 512MB would be more useful in my situation?
Sorry for the 1000 questions, i'm getting about 50 memory limit errors per hour right now, and im just trying to find something to ease them a little.

TOTAL_MEMORY is the size of memory addressable by C. If a program tries to malloc enough that it requires more than that, it will fail (unless you build with memory growth).

An application can use more memory than just the memory addressable by C. For example, files in the filesystem might be stored separately (there are some options there).

I've got emscripten running on an Ubuntu 14.04 box. When you say " recompiling with ALLOW_MEMORY_GROWTH flag" can you expand on that a little? where do I set this? I installed using apt and im able to build the JS lib, im just not sure where that flag is.

give emcc the extra args -s ALLOW_MEMORY_GROWTH=1

Getting there...

I edited the Makefile, changed the optimized line to:

optimized: EMFLAGS= --memory-init-file 0 --closure 1 -O3  -s INLINING_LIMIT=50 -s ALLOW_MEMORY_GROWTH=1

The build fails with:
Cannot enable ALLOW_MEMORY_GROWTH with asm.js, build with -s ASM_JS=0 if you need a growable heap

So, I edit the line again and added -s ASM_JS=0 and this time it finishes.

The JS looks ok, but the browser throws an error when using it:
Uncaught TypeError: Cannot read property 'Ha' of undefined

The line looks minified, and not terribly useful, but the section is: [bc,za.Ha,za.Ec,za.Xa]

I also noticed that the JS file is about 930Kb, which is smaller than the one in your repo. So im wondering what is missing.

Any ideas? Looks like something is missing. Is there something else I should be doing rather than just editing this file and calling make ?

What version of emscripten is that with? Disabling asm.js should break (with a clear error) on all recent versions.

emcc -v
emcc (Emscripten GCC-like replacement + linker emulating GNU ld ) 1.10.0
Ubuntu clang version 3.3-16ubuntu1 (branches/release_33) (based on LLVM 3.3)
Target: x86_64-pc-linux-gnu
Thread model: posix

Doing install latest failed for me, with the following error although I was able build an earlier version (a stackoverflow suggestion)

lude/llvm/Support/DataTypes.h:77:3: error: #error "Don't have a definition for uint64_t on this platform"
 # error "Don't have a definition for uint64_t on this platform"
   ^

On Ubuntu 14-04

1.10.0 is indeed quite old - I guess that's from the ubuntu repos? Stable is 1.34.0.

I remember we had differences in how memory growth works in older versions, it used to disable asm.js. But that is no longer the case in recent versions. Not sure if that is the cause of your issues, but could be.

Can try to build without closure and with -g to get a readable build, might indicate what is wrong on that minified line.

Hi,

I am trying to compile with this option as well -s ALLOW_MEMORY_GROWTH=1. Still installing emcc, but I want to make sure we shouldn't add ASM_JS=0 when compiling?

thanks for this code, it's super useful for me, only need to get this memory option working and it would be perfect.

Definitely cannot add ASM_JS=0 on recent emscripten versions. We have been supporting memory growth with asm.js for a while now.

Would it be possible to just add a compiled version of the latest sql.js with ALLOW_MEMORY_GROWTH bit set added to this repo ? or even just a once off that I can download from somewhere? I’m not 100% sure it’s going to solve my problem, and I've wasted enough of your time trying to get my build environment up and running. I appreciate the help, I don’t want to keep bugging you though.

I compiled with adding only -s ALLOW_MEMORY_GROWTH=1 and it seems it worked for me:

https://raw.githubusercontent.com/lpantano/seqclusterViz/master/js/sql.js

thanks for the help!

Thanks @lpantano!
That link is the re-compiled version with ALLOW_MEMORY_GROWTH set right? I'll try it out of so

yes, it is. I compiled and pushed to the repo is using sql.js hope you have the same luck!

Thanks @lpantano , you just saved me a few hours of hassle :)

Thanks @lpantano!

Might be nice to add a build in this repo too, so people can easily find it, if @lovasoa is ok with that.

Yes, good idea! It's not the first time someone comes up with this problem, and not everyone has the courage to build SQL.js on his own.

Le 21 juillet 2015 02:59:15 GMT+02:00, Alon Zakai notifications@github.com a écrit :

Thanks @lpantano!

Might be nice to add a build in this repo too, so people can easily
find it, if @lovasoa is ok with that.

---

Reply to this email directly or view it on GitHub:
https://github.com/kripken/sql.js/issues/113#issuecomment-123114795

I'm getting horrible insert throughput, like only 100/s. Does this sound right?

Please open a new issue, and give more details.

For those googling the Cannot enlarge memory arrays problem and cannot/don't want to touch the sql.js file:

Append Module = { TOTAL_MEMORY: X }; to the js file before the require sql.js line.

Substitute X with the number of bytes required.

For those googling the Cannot enlarge memory arrays problem and cannot/don't want to touch the sql.js file:

Append Module = { TOTAL_MEMORY: X }; to the js file before the require sql.js line.

Substitute X with the number of bytes required.

@duffrind this raised error "'Module' is not defined"