[Kinesis] add Scala kinesis-mock build behind feature flag #12559

gregfurman · 2025-04-25T13:39:35Z

Motivation

We've observed the NodeJS build of kinesis mock to encounter performance issues at higher volumes of requests -- namely with larger payload sizes.

This PR adds the original Scala build as an alternative engine for our Kinesis server behind the KINESIS_MOCK_PROVIDER_ENGINE flag.

Notes

Relevant upstream issue: [Feature Request] Include Scala build as a release artifact etspaceman/kinesis-mock#1082
Ext run with flag enabled: https://github.com/localstack/localstack-ext/actions/runs/14646658785

Changes

Functionality

Splits up the KinesisMockServer to include a KinesisMockScalaServer and KinesisMockNodeServer
KinesisMockPackage can now either use the KinesisMockScalaPackageInstaller (downloads from GitHub) or KinesisMockNodePackageInstaller (downloads from NPM)

Config

Adds a KINESIS_MOCK_PROVIDER_ENGINE flag for switching to Scala build of Kinesis Mock. Valid values are node or scala -- where empty or invalid values will always default to node.
KINESIS_MOCK_MAXIMUM_HEAP_SIZE sets the maximum Java heap size corresponding to the '-Xmx' flag
KINESIS_MOCK_INITIAL_HEAP_SIZE sets the initial Java heap size corresponding to the '-Xms' flag

Testing

Creates a TestKinesisMockScala(TestKinesis) subclass that runs all tests in tests/aws/services/kinesis/test_kinesis.py::TestKinesis but with the configuration value monkeypatched to use the Scala build of Kinesis mock instead of NodeJS.

github-actions · 2025-04-25T13:51:36Z

S3 Image Test Results (AMD64 / ARM64)

2 files 2 suites 8m 41s ⏱️
488 tests 438 ✅ 50 💤 0 ❌
976 runs 876 ✅ 100 💤 0 ❌

Results for commit a089a68.

♻️ This comment has been updated with latest results.

github-actions · 2025-04-25T15:23:04Z

LocalStack Community integration with Pro

2 files ± 0 2 suites ±0 1h 43m 32s ⏱️ + 1m 3s
4 441 tests +16 4 058 ✅ +16 383 💤 ±0 0 ❌ ±0
4 443 runs +16 4 058 ✅ +16 385 💤 ±0 0 ❌ ±0

Results for commit a089a68. ± Comparison against base commit b961fee.

♻️ This comment has been updated with latest results.

dfangl · 2025-04-28T09:57:43Z

localstack-core/localstack/services/kinesis/kinesis_mock_server.py

+            "-XX:+UseG1GC",
+            "-XX:MaxGCPauseMillis=500",
+            "-XX:+UseGCOverheadLimit",
+            "-XX:+ExplicitGCInvokesConcurrent",
+            "-XX:+HeapDumpOnOutOfMemoryError",
+            "-XX:+ExitOnOutOfMemoryError",


Do we really need to set all these flags? Do they improve performance in any way? Especially the heap dump seems like something our customers do not want, and G1GC is generally the default GC in all recent java versions.

The MaxGCPauseMillis was for performance reasons and the ExitOnOutOfMemoryError was because I didn't want the server to stall when encountering an OOM.

Will remove:

UseG1GC since I was unaware it was the default (as per your comment)

UseGCOverheadLimit since it does nothing with G1GC

HeapDumpOnOutOfMemoryError since it'll spam the logs quite a bit

dfangl · 2025-04-28T10:00:52Z

localstack-core/localstack/services/kinesis/packages.py

+    @cached_property
+    def engine(self) -> KinesisMockEngine:
+        return KinesisMockEngine(config.KINESIS_MOCK_PROVIDER_ENGINE)
+


Why do we need this?

To determine which Kinesis mock backend to run:

localstack/localstack-core/localstack/services/kinesis/kinesis_mock_server.py

Lines 208 to 225 in 4eed69f

if kinesismock_package.engine == KinesisMockEngine.SCALA:

return KinesisMockScalaServer(

port=port,

exe_path=kinesis_mock_path,

log_level=log_level,

latency=latency,

data_dir=persist_path,

account_id=account_id,

)

return KinesisMockNodeServer(

port=port,

exe_path=kinesis_mock_path,

log_level=log_level,

latency=latency,

data_dir=persist_path,

account_id=account_id,

)

In this case - I am not that happy with the control here. The Servers should control which package is installed, not the package which server is used, in my opinion.
Could simplify this a bit, by having two packages, and installing the right one depending on the configuration, and then use the right server?

Yeah sure happy to have 2 packages 👍

@dfangl How should we go about configuring the plugin for the LPM?

Should we have a seperate kinesis-mock and kinesis-mock-scala plugin or should this implementation be unified to cater for both node + scala builds i.e

@package(name="kinesis-mock") def kinesismock_package() -> Package: from localstack.services.kinesis.packages import kinesismock_package return kinesismock_package @package(name="kinesis-mock-scala") def kinesismock_scala_package() -> Package: from localstack.services.kinesis.packages import kinesismock_scala_package return kinesismock_scala_package

OR

@package(name="kinesis-mock") def kinesismock_package() -> Package: from localstack.services.kinesis.packages import ( KinesisMockEngine, kinesismock_package, kinesismock_scala_package, ) if KinesisMockEngine(config.KINESIS_MOCK_PROVIDER_ENGINE) == KinesisMockEngine.SCALA: return kinesismock_scala_package return kinesismock_package

~~@dfangl FYI I spoke to Alex re: the above and we went with the second approach ☝️~~ Nvm: going to comment on your latest review

dfangl

LGTM! Way cleaner than before in my opinion 👍

dfangl · 2025-05-16T11:40:35Z

localstack-core/localstack/services/kinesis/packages.py

+    def _missing_(cls, value: str | Any) -> str:
+        # default to 'node' if invalid enum
+        if not isinstance(value, str):
+            return cls(cls.NODE)
+        return cls.__members__.get(value.upper(), cls.NODE)


Nit: Not entirely happy with this fallback logic here - it is effectively unused, as we only ever check if it is scala anyway, and we would silently accept obviously wrong values. Also, if we ever switch the default, we have to do it in multiple places. Nothing to hold back the merge, but I think we could potentially make this easier.

Yeah I suppose this is unideal but I wanted to make sure (since we're feature-flagging this strategy) that nothing gets broken if incorrect values are used. Definitely erring on the defensive side.

What have we done in past for situations like this?

Usually we would fallback, but at least with a warning in the log.

localstack-core/localstack/services/kinesis/plugins.py

gregfurman added area: performance Make LocalStack go rocket-fast aws:kinesis Amazon Kinesis semver: minor Non-breaking changes which can be included in minor releases, but not in patch releases labels Apr 25, 2025

gregfurman added this to the 4.4 milestone Apr 25, 2025

gregfurman requested a review from dfangl April 25, 2025 13:39

gregfurman self-assigned this Apr 25, 2025

gregfurman force-pushed the scala/kinesis-mock branch from 65c9b9a to a400993 Compare April 25, 2025 14:11

gregfurman force-pushed the scala/kinesis-mock branch from a400993 to 4eed69f Compare April 25, 2025 16:25

gregfurman marked this pull request as ready for review April 25, 2025 17:22

gregfurman requested review from alexrashed and thrau as code owners April 25, 2025 17:22

dfangl reviewed Apr 28, 2025

View reviewed changes

gregfurman modified the milestones: 4.4, 4.5 May 6, 2025

gregfurman added 2 commits May 12, 2025 17:36

[Kinesis] add Scala kinesis-mock build behind feature flag

307b7ed

Address comments

cc634d2

gregfurman force-pushed the scala/kinesis-mock branch from 4eed69f to cc634d2 Compare May 12, 2025 15:36

gregfurman requested a review from dfangl May 12, 2025 15:38

Unify packages

a089a68

dfangl approved these changes May 16, 2025

View reviewed changes

gregfurman merged commit 9990b6f into master May 16, 2025
39 checks passed

gregfurman deleted the scala/kinesis-mock branch May 16, 2025 13:41

gregfurman mentioned this pull request Jun 5, 2025

add(kinesis): Include new Scala engine and performance section localstack/docs#1786

Open

	if kinesismock_package.engine == KinesisMockEngine.SCALA:
	return KinesisMockScalaServer(
	port=port,
	exe_path=kinesis_mock_path,
	log_level=log_level,
	latency=latency,
	data_dir=persist_path,
	account_id=account_id,
	)

	return KinesisMockNodeServer(
	port=port,
	exe_path=kinesis_mock_path,
	log_level=log_level,
	latency=latency,
	data_dir=persist_path,
	account_id=account_id,
	)

Uh oh!

[Kinesis] add Scala kinesis-mock build behind feature flag #12559

[Kinesis] add Scala kinesis-mock build behind feature flag #12559

Uh oh!

Conversation

gregfurman commented Apr 25, 2025

Motivation

Notes

Changes

Functionality

Config

Testing

Uh oh!

github-actions bot commented Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

S3 Image Test Results (AMD64 / ARM64)

Uh oh!

github-actions bot commented Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

LocalStack Community integration with Pro

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gregfurman May 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gregfurman May 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dfangl left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Apr 25, 2025 •

edited

Loading

github-actions bot commented Apr 25, 2025 •

edited

Loading

gregfurman May 13, 2025 •

edited

Loading

gregfurman May 16, 2025 •

edited

Loading