Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add pre builds for aws lambda #155

Closed
julianhille opened this issue May 9, 2022 · 34 comments
Closed

add pre builds for aws lambda #155

julianhille opened this issue May 9, 2022 · 34 comments

Comments

@julianhille
Copy link
Owner

aws lambda linux is not compatible to a normal linux build

@vamshi9
Copy link

vamshi9 commented May 20, 2022

Hey @julianhille Any progress on this?

@julianhille
Copy link
Owner Author

I'm not even sure if this is really needed. It builds and would just be a matter of hours to release it. Are you able to test it?

@vamshi9
Copy link

vamshi9 commented May 21, 2022

I am actually not able to test it as I am currently facing an issue with puppeteer on lambda, was not able to launch chrome during run time.

@JacksonWolff
Copy link

Any updates on this? facing a few issues to make it to work on a aws lambda.

@julianhille
Copy link
Owner Author

julianhille commented Jun 17, 2022

Ok this really is still an issue? Thanks for telling. I need to decide what we need to fake as there is no such thing as a "special aws" Linux.
So we need to decide to fake a platform, clib or architecture but the easiest would be to invent a new platform name.

Would you be able to test a release candidate / beta / pre-release or so if I'd create one?

@kevinvoduy
Copy link

I'd be happy to help in any way as I'd like support for this as well

@julianhille
Copy link
Owner Author

#156
That PR includes some readme changes
And here: https://github.com/julianhille/MuhammaraJS/actions/runs/2527354590 you could download some pre builts.

@julianhille
Copy link
Owner Author

please have a look and test if the prebuilts work and the if the readme is enough to understand how to use it on/with aws lambda.

@JacksonWolff
Copy link

Will do that tomorrow for sure. Thanks @julianhille for the effort on this.

@julianhille
Copy link
Owner Author

@vamshi9 and @kevinvoduy you both could check if that helps

@kevinvoduy
Copy link

kevinvoduy commented Jun 21, 2022

I was able to verify that build node-v72-linux-x64-glibc worked for my lambda setup.

Steps to reproduce:

  • create a nodejs directory (later to be uploaded as lambda layer)
  • the directory should contain node_modules and package.json
  • install muhammara and replace the muhammara.binding with one provided in https://github.com/julianhille/MuhammaraJS/actions/runs/2527354590 (I used node-v72-linux-x64-glibc)
  • zip the nodejs folder and upload as a lambda layer
  • add the layer to a lambda and test

No longer seeing invalid elf header error

@julianhille
Copy link
Owner Author

That is interesting as this file has not changed. The only files having changed are, according to the readme and the code I changed, node-v72-amz-lambda-linux-x64

@julianhille
Copy link
Owner Author

@kevinvoduy is it possible to chat with you somehow / somewhere to find this issue a bit faster?

If not:

  • are you building the package (the zip) to upload to aws on mac or windows?
  • if so: are you using specific commands to get the correct target platform?

@kevinvoduy
Copy link

Hey Julian, I'll be checking for messages from here for the rest of the day. I'm zipping the package on mac. I first replace the muhamarra.node (node-v72-linux-x64-glibc) then zip then upload to lambda. It worked on a test lambda that I had set up. You later called out that I was using the wrong version, so I repeated the process but with node-v72-amz-lambda-linux-x64. That appears to be working as well.

I didn't run any specific commands. Right-click on the nodejs folder and "Compress". I can spin up a ubuntu docker container and zip it there instead?

@julianhille
Copy link
Owner Author

Ok so you are just execute npm install (or yarn or whatever your PKG manager is) right?

@kevinvoduy
Copy link

correct

@julianhille
Copy link
Owner Author

Ok. There is the issue. Muhammara, along with a lot of other modules which needs compiling or are in any case specific for a platform, can't be installed on platform x (windows for ex) and then transfer the files over to another platform.

You need to tell npm what platform you want to install to.

@kevinvoduy
Copy link

I'll look into that and provide an update.

@julianhille
Copy link
Owner Author

julianhille commented Jun 22, 2022

You could use one of the options provided here.
BUT do not switch platform to amz-linux but just use linux. What you are actually doing is option 3.

@julianhille
Copy link
Owner Author

EXTRA_NODE_PRE_GYP_FLAGS="--target_platform=linux" npm install

or after npm install:
npx node-pre-gyp install --update-binary --target-platform=linux

@julianhille
Copy link
Owner Author

julianhille commented Jun 22, 2022

A third option is more complex but would be the most correct variant: take a docker container from aws lambda for your node version as base.
Mount your app into it and execute npm installation there and then package / zip up what is needed. That'd be the most complete and correct way of building the correct dependencies for your aws deployment.

@Joshuabaker2
Copy link

I'm attempting to get this working but not having any luck. I'm on Node 16.x for AWS, so I grabbed the binding from node-v93-amz-lambda-linux-x64-glibc from the github actions build. Replaced the muhammara.node in the package.json with that one and zipped the lambda and uploaded manually to AWS. I'm not getting any runtime errors, but I am getting a broken PDF back. Running it locally on my machine produces a nicely working pdf with the same inputs.

I'll try your docker container suggestion next.

@julianhille
Copy link
Owner Author

That should kinda end up the same. Could you attach your zipped package or at least the node modules dir before unloading?

@Joshuabaker2
Copy link

Sure thing, here's my node_modules (it only contains muhammara)
node_modules.zip

If you'd like, I could make a dummy lambda that processes a pdf from a base64-encoded body POST if you have access to aws lambda to test it as well.

@julianhille
Copy link
Owner Author

ok this should work on aws with node v16. are you using v16?

@julianhille
Copy link
Owner Author

Yeah create a tiny project and upload it which just returns a somehow edited pdf.

@julianhille
Copy link
Owner Author

I'm sorry now I get it. The pdf is broken without an error by muhammara. Could you also attach the pdf? And what calls against muhammara are you doing? Sometime ago when it was hummus I once fixed an pdf Font issue with a segfault in the Linux part of hummus.
Are you perhaps streaming the response while editing it? Guess so, because lambda has mo real storage. Could you provide call order? Feels like it is a segfault cause of accessing some missing shared libs.

@Joshuabaker2
Copy link

Sure thing, here's the lambda. The code is un-minified so you should be able to read through it, i've also included the successfully output pdf run on my m1 macbook pro.

If you are uploading it to lambda, the handler is src/lambda/http/doc2.handler

broken-pdf.pdf

sample-docs.zip

@Joshuabaker2
Copy link

Just as an additional piece of data, I also tried running the same code on aws lambda with node14 and the node14 bindings and same problem.

@julianhille
Copy link
Owner Author

i'm not sure if you want the pdf or the domain exposed thats why i do not upload the correct pdf, but the file is kinda fine. Its just base64 encoded.

cat broken-pdf.pdf | base64 -d > fine.pdf so it must be something with your handler stating isBae64Encoded and not acting accordingly on it.

@Joshuabaker2
Copy link

Interesting.... thank you for this clue! I'll report when I figure out how to make this work.

@Joshuabaker2
Copy link

Joshuabaker2 commented Jun 23, 2022

So the trick that got it to work... makes me feel quite silly. I had to set my request header to include Accept: application/pdf. I had Accept: */* but that wasn't sufficient. Including that specific Accept header got it to work. Thanks for your help @julianhille!

@Joshuabaker2
Copy link

Also just as a note, both the standard linux and the specially-made aws binding work fine.

@julianhille
Copy link
Owner Author

So the trick that got it to work... makes me feel quite silly. I had to set my request header to include Accept: application/pdf. I had Accept: */* but that wasn't sufficient. Including that specific Accept header got it to work. Thanks for your help @julianhille!

Thank you for reporting back, great to hear you could solve it.
You can always buy me a coffee or cuba libre

Also just as a note, both the standard linux and the specially-made aws binding work fine.

Thank you so much for reporting this too.
I kinda expected that but was not really sure. I know that we had to compile some libs for python bindings on the aws specific aws lambda docker to make it work, so i gave it a try here. But until now i did not see anyone really having an issue with the normal linux lib on aws.

So im closing this for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants