Open Radar

 
AWS metadata service endpoint 169.254.169.254 is problematic on CI build machines due to Local Network Privacy restrictions

Description:

Sequoia’s Local Network Privacy feature is problematic for customers who run EC2 Mac instances in Amazon AWS. It makes it difficult to adopt basic functionality from working in the context of continuous integration (CI) builds and in general the configuration of the host.


Background:

In AWS, it is commonplace to use the IMDS (Instance Metadata Service) to securely obtain a “role” to access other services and features within AWS. These can include (but are definitely not limited to):

* Configuration of a CI agent, including setting additional metadata to help describe its capabilities (Buildkite, Github Actions, Gitlab, etc.)
* Fetching credentials from Secrets Manager
* Downloading / uploading to S3 buckets
* Invoking lambda functions
* Using IAM to access Instance Identity documents to allow other called services to attest their identity


To briefly explain how this works: the host (running in AWS, in this particular case a Mac but this is how all EC2 instances work) has access to a local link IP address at 169.254.169.254, which accepts a TCP connection on port 80 to obtain a token for temporary credentials, which it can use to perform actions like I listed above. The Mac has access to this IP via AWS’s Thunderbolt ethernet adapter connected to the Mac.

Where LNP comes into play: this feature includes this local link IP address in the list of special address ranges which are blocked by the LNP system, and therefore require manual approval. 



Impact:

This limitation makes it particularly challenging for typical workloads for CI (Continuous Integration) agents, where the "runner" of all commands ultimately stems from a single agent process that is connected to an orchestrator system (GitHub, Jenkins, Buildkite, Gitlab, etc.) , if running in the context of a LaunchAgent.

Switching to a LaunchDaemon, while it may help with the “permission” of allowing the local network connection, poses other problems. Build and test jobs have issues working reliably when running as a LaunchDaemon at the loginwindow, without access to a regular Aqua user session context. For example, Keychain access for accessing credentials and codesigning, lack of access to the window manager or ability to launch apps in a desktop session, etc.


AWS EC2 Mac is commonly used by companies who run continuous integration clusters of Mac hardware for building and testing iOS applications. My employer runs roughly 750 EC2 Mac machines and use them to build and test Cash App, the Square Point of Sale family of apps, Bitkey, and several other applications. We rely on automation to be able to reliably configure our build machines, and are able to install and configure what we need via automated scripts and Ansible. However, because we’re not able to automatically configure any tools as “trusted” for Local Network Privacy, these steps must be done manually. Moreover, it is challenging to _update_ any of these tools as we must revert to either a new or previously-saved machine, and re-perform the approval. We run automatic builds of our CI machines daily in order to ensure that our provisioning system is stable and easy to make changes to.

The list of services and features I listed in the Background section at the top of this feedback is a list of features we actively use in our CI workloads today – they are not hypothetical use-cases. Every developer-submitted and release build of our Square and Cash App compilation and testing processes require these components in order to securely access credentials and authenticate to other systems which are depended on as part of our build.


The broader downstream impact of this issue, is that when the spring Xcode release requires Sequoia, we will not be ready to support it on our build machines. I’m aware of other large software companies with very well-known iOS apps who are in a similar position. In talking with other companies who maintain Mac continuous integration clusters, this seems to be a common problem. I believe AWS is also already engaged with Apple in this issue.


Request:

I propose that an exception be made in this LNP policy for the 169.254.169.254 IP address specifically. I understand the intent of the LNP system in order to improve security for Mac customers, avoiding having unintended (and potentially malicious) local network traffic. The IP address of 169.254.169.254, however, seems uniquely used for AWS and other cloud providers, and the usage pattern for running Mac in a hosted cloud is very different than that of a customer running their own personal or corporate laptop or desktop Mac.

Alternatively, if there is some mechanism to generate a trusted manifest of LNP processes and then configure a different machine with those processes pre-trusted (even if only for 169.254.169.254) that also could potentially work.


Apple has publicly recommended AWS EC2 Mac in this news briefing (https://developer.apple.com/news/?id=swfemvx0): "For the first time, you can easily set up and deploy macOS workloads natively within AWS, and take advantage of its flexibility and scalability to add more compute capacity." With the LNP restrictions in macOS Sequoia, however, there are now significant hurdles to being able to take advantage of AWS's features for security and automation.



References:
* Basics on AWS IMDS: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
* Understanding Local Network Privacy TN: https://developer.apple.com/documentation/technotes/tn3179-understanding-local-network-privacy