Reduce requests necessary for lambda log publishing to cloudwatch logs #10234
Conversation
| logs = log_item.logs.splitlines() | ||
| # until we have a better way to have timestamps, log events have the same time for a single invocation | ||
| log_events = [ | ||
| {"timestamp": int(time.time() * 1000), "message": log_line} for log_line in logs |
There was a problem hiding this comment.
Could that behavioral change mess up the log order in CloudWatch?
Compared to the previous heuristic:
simple heuristic: assume log lines were emitted in regular intervals
another reason to tackle structured logging and improved logging soon ...
There was a problem hiding this comment.
We will probably be fine, as it should not get reordered based on the timestamp in our codebase. Definitely something to look out for, and fix as soon as possible.
There was a problem hiding this comment.
😬 🤞
Not in our Python code, but it could happen when querying from sqlite with CloudWatch v2 🤷
For example in localstack/services/cloudwatch/cloudwatch_database_helper.py, if we ScanBy.TimestampDescending 🙈
There was a problem hiding this comment.
I could increase it in 1ms intervals to be sure :)
There was a problem hiding this comment.
But reordering Events with the same timestamp would be a parity violation anyway, so not sure how much sense it makes.
There was a problem hiding this comment.
yeah, but we never had this parity
the closest we have to parity is event ordering (through some random heuristic)
There was a problem hiding this comment.
guess it doesn't matter too much if we fix it soonish 🔮
Motivation
We currently make 3 requests to CloudWatch logs for every lambda invocation:
CreateLogGroup(will fail for all but the first invoke for that function)CreateLogStream(will fail for all but the first invoke for that execution environment)PutLogEventsWe can drastically reduce the calls, and also load on the gateway, if we only execute the first two calls if
PutLogEventsfails.If it fails, it equals to 4 calls (failed
PutLogEvents, thenCreateLogGroup, thenCreateLogStream), but then only one call for every invocation (for the same environment) afterwards.Compared to 3 calls for every single invocation right now. There is also a lot of logic in
store_cloudwatch_logswhich is not necessary anymore with the new lambda provider.Changes