diff --git a/Makefile b/Makefile index c67d2b8..fde661b 100644 --- a/Makefile +++ b/Makefile @@ -1,13 +1,15 @@ -serve: - python -m http.server --directory=_site 8888 +all: build update-book serve build: - ./generate-html + ./generate-html.py + +serve: + python -m http.server 8899 watch-build: - ls **/*.md **/*.html *.py | entr ./generate-html.py + ls **/*.md **/*.html **/*.xml *.py | entr ./generate-html.py update-book: ## assumes book repo is at ../book cd ../book && make html ./copy-and-fix-book-html.py - rsync -a -v ../book/images/ _site/book/images/ + rsync -a -v ../book/images/ ./book/images/ diff --git a/docs/blog/2017-09-07-introducing-command-handler.html b/blog/2017-09-07-introducing-command-handler.html similarity index 92% rename from docs/blog/2017-09-07-introducing-command-handler.html rename to blog/2017-09-07-introducing-command-handler.html index 57dce06..f1b6518 100644 --- a/docs/blog/2017-09-07-introducing-command-handler.html +++ b/blog/2017-09-07-introducing-command-handler.html @@ -2,25 +2,35 @@ - - - - + + + Introducing Command Handler + + + + + + + + + + + -
+
-
diff --git a/blog/2017-09-08-repository-and-unit-of-work-pattern-in-python.html b/blog/2017-09-08-repository-and-unit-of-work-pattern-in-python.html new file mode 100644 index 0000000..64815c9 --- /dev/null +++ b/blog/2017-09-08-repository-and-unit-of-work-pattern-in-python.html @@ -0,0 +1,370 @@ + + + + + + + Repository and Unit of Work Pattern + + + + + + + + + + + + + + + + + + + +
+ + + +
+ +

Repository and Unit of Work Pattern

+

by Bob, 2017-09-08

+ + + + + +
+

In the previous part +(Introducing Command Handler) +of this series we built a toy system that could add a new Issue to an IssueLog, but +had no real behaviour of its own, and would lose its data every time the +application restarted. We’re going to extend it a little by introducing some +patterns for persistent data access, and talk a little more about the ideas +underlying ports and adapters architectures. To recap, we’re abiding by three +principles:

+
    +
  1. Clearly define the boundaries of our use cases.
  2. +
  3. Depend on abstractions, not on concrete implementation.
  4. +
  5. Identify glue code as distinct from domain logic and put it into its own + layer.
  6. +
+

In our command handler, we wrote the following code:

+
reporter = IssueReporter(cmd.reporter_name, cmd.reporter_email)
+issue = Issue(reporter, cmd.problem_description)
+issue_log.add(issue)
+
+ +

The IssueLog is a term from our conversation with the domain expert. It’s the +place that they record the list of all issues. This is part of the jargon used +by our customers, and so it clearly belongs in the domain, but it’s also the +ideal abstraction for a data store. How can we modify the code so that our newly +created Issue will be persisted? We don’t want our IssueLog to depend on the +database, because that’s a violation of principle #2. This is the question that +leads us to the ports & adapters architecture.

+

In a ports and adapters architecture, we build a pure domain that exposes ports. +A port is a way for data to get into, or out of, the domain model. In this +system, the IssueLog is a port. Ports are connected to the external world by +Adapters. In the previous code sample, the FakeIssueLog is an adapter: it +provides a service to the system by implementing an interface.

+

Let’s use a real-world analogy. Imagine we have a circuit that detects current +over some threshold. If the threshold is reached, the circuit outputs a signal. +Into our circuit we attach two ports, one for current in, and one for current +out. The input and output channels are part of our circuit: without them, the +circuit is useless.

+
class ThresholdDetectionCircuit:
+
+    arbitrary_threshold = 4
+
+    def __init__(self, input: ReadablePort, output: WriteablePort):
+        self.input = input
+        self.output = output
+
+    def read_from_input(self):
+        next_value = self.input.read()
+        if next_value > self.arbitrary_threshold:
+            self.output.write(1)
+
+ +

Because we had the great foresight to use standardised ports, we can plug any +number of different devices into our circuit. For example, we could attach a +light-detector to the input and a buzzer to the output, or we could attach a +dial to the input, and a light to the output, and so on.

+
class LightDetector(ReadablePort):
+    def read(self):
+        return self.get_light_amplitude()
+
+class Buzzer(WriteablePort):
+    def write(self, value):
+        if value > 0:
+            self.make_infuriating_noise()
+
+
+class Dial(ReadablePort):
+    def read(self):
+        return self.current_value
+
+class Light(self):
+    def write(self, value):
+        if value > 0:
+            self.on = True
+        else:
+            self.on = False
+
+ +

Considered in isolation, this is just an example of good OO practice: we are +extending our system through composition. What makes this a ports-and-adapters +architecture is the idea that there is an internal world consisting of the +domain model (our ThresholdDetectionCircuit), and an external world that drives +the domain model through well-defined ports. How does all of this relate to +databases?

+
from SqlAlchemy import Session
+
+class SqlAlchemyIssueLog (IssueLog):
+
+    def __init__(self, session: Session):
+        self.session = session
+
+    def add(self, issue):
+        self.session.add(issue)
+
+
+class TextFileIssueLog (IssueLog):
+
+    def __init__(self, path):
+        self.path = path
+
+    def add(self, issue):
+        with open(self.path, 'w') as f:
+            json.dump(f)
+
+ +

By analogy to our circuit example, the IssueLog is a WriteablePort - it’s a way +for us to get data out of the system. SqlAlchemy and the file system are two +types of adapter that we can plug in, just like the Buzzer or Light classes. In +fact, the IssueLog is an instance of a common design pattern: it’s a Repository +[https://martinfowler.com/eaaCatalog/repository.html]. A repository is an object +that hides the details of persistent storage by presenting us with an interface +that looks like a collection. We should be able to add new things to the +repository, and get things out of the repository, and that’s essentially it.

+

Let’s look at a simple repository pattern.

+
class FooRepository:
+    def __init__(self, db_session):
+        self.session = db_session
+
+    def add_new_item(self, item):
+        self.db_session.add(item)
+
+    def get_item(self, id):
+        return self.db_session.get(Foo, id)
+
+    def find_foos_by_latitude(self, latitude):
+        return self.session.query(Foo).\
+                filter(foo.latitude == latitude)
+
+ +

We expose a few methods, one to add new items, one to get items by their id, and +a third to find items by some criterion. This FooRepository is using a +SqlAlchemy session +[http://docs.sqlalchemy.org/en/latest/orm/session_basics.html] object, so it’s +part of our Adapter layer. We could define a different adapter for use in unit +tests.

+
class FooRepository:
+    def __init__(self, db_session):
+        self.items = []
+
+    def add_new_item(self, item):
+        self.items.append(item)
+
+    def get_item(self, id):
+        return next((item for item in self.items 
+                          if item.id == id))
+
+    def find_foos_by_latitude(self, latitude):
+        return (item for item in self.items
+                     if item.latitude == latitude)
+
+ +

This adapter works just the same as the one backed by a real database, but does +so without any external state. This allows us to test our code without resorting +to Setup/Teardown scripts on our database, or monkey patching our ORM to return +hard-coded values. We just plug a different adapter into the existing port. As +with the ReadablePort and WriteablePort, the simplicity of this interface makes +it simple for us to plug in different implementations.

+

The repository gives us read/write access to objects in our data store, and is +commonly used with another pattern, the Unit of Work +[https://martinfowler.com/eaaCatalog/unitOfWork.html]. A unit of work represents +a bunch of things that all have to happen together. It usually allows us to +cache objects in memory for the lifetime of a request so that we don’t need to +make repeated calls to the database. A unit of work is responsible for doing +dirty checks on our objects, and flushing any changes to state at the end of a +request.

+

What does a unit of work look like?

+
class SqlAlchemyUnitOfWorkManager(UnitOfWorkManager):
+    """The Unit of work manager returns a new unit of work. 
+       Our UOW is backed by a sql alchemy session whose 
+       lifetime can be scoped to a web request, or a 
+       long-lived background job."""
+    def __init__(self, session_maker):
+        self.session_maker = session_maker
+
+    def start(self):
+        return SqlAlchemyUnitOfWork(self.session_maker)
+
+
+class SqlAlchemyUnitOfWork(UnitOfWork):
+    """The unit of work captures the idea of a set of things that
+       need to happen together. 
+
+       Usually, in a relational database, 
+       one unit of work == one database transaction."""
+
+    def __init__(self, sessionfactory):
+        self.sessionfactory = sessionfactory
+
+    def __enter__(self):
+        self.session = self.sessionfactory()
+        return self
+
+    def __exit__(self, type, value, traceback):
+        self.session.close()
+
+    def commit(self):
+        self.session.commit()
+
+    def rollback(self):
+        self.session.rollback()
+
+    # I tend to put my repositories onto my UOW
+    # for convenient access. 
+    @property
+    def issues(self):
+        return IssueRepository(self.session)
+
+ +

This code is taken from a current production system - the code to implement +these patterns really isn’t complex. The only thing missing here is some logging +and error handling in the commit method. Our unit-of-work manager creates a new +unit-of-work, or gives us an existing one depending on how we’ve configured +SqlAlchemy. The unit of work itself is just a thin layer over the top of +SqlAlchemy that gives us explicit rollback and commit points. Let’s revisit our +first command handler and see how we might use these patterns together.

+
class ReportIssueHandler:
+    def __init__(self, uowm:UnitOfWorkManager):
+        self.uowm = uowm
+
+    def handle(self, cmd):
+        with self.uowm.start() as unit_of_work:
+            reporter = IssueReporter(cmd.reporter_name, cmd.reporter_email)
+            issue = Issue(reporter, cmd.problem_description)
+            unit_of_work.issues.add(issue)
+            unit_of_work.commit()
+
+ +

Our command handler looks more or less the same, except that it’s now +responsible for starting a unit-of-work, and committing the unit-of-work when it +has finished. This is in keeping with our rule #1 - we will clearly define the +beginning and end of use cases. We know for a fact that only one object is being +loaded and modified here, and our database transaction is kept short. Our +handler depends on an abstraction - the UnitOfWorkManager, and doesn’t care if +that’s a test-double or a SqlAlchemy session, so that’s rule #2 covered. Lastly, +this code is painfully boring because it’s just glue. We’re moving all the dull +glue out to the edges of our system so that we can write our domain model in any +way that we like: rule #3 observed.

+

The code sample for this part +[https://github.com/bobthemighty/blog-code-samples/tree/master/ports-and-adapters/02] + adds a couple of new packages - one for slow tests +[http://pycon-2012-notes.readthedocs.io/en/latest/fast_tests_slow_tests.html] +(tests that go over a network, or to a real file system), and one for our +adapters. We haven’t added any new features yet, but we’ve added a test that +shows we can insert an Issue into a sqlite database through our command handler +and unit of work. Notice that all of the ORM code is in one module +(issues.adapters.orm) and that it depends on our domain model, not the other way +around. Our domain objects don’t inherit from SqlAlchemy’s declarative base. +We’re beginning to get some sense of what it means to have the domain on the +“inside” of a system, and the infrastructural code on the outside.

+

Our unit test has been updated to use a unit of work, and we can now test that +we insert an issue into our issue log, and commit the unit of work, without +having a dependency on any actual implementation details. We could completely +delete SqlAlchemy from our code base, and our unit tests would continue to work, +because we have a pure domain model and we expose abstract ports from our +service layer.

+
class When_reporting_an_issue:
+
+    def given_an_empty_unit_of_work(self):
+        self.uow = FakeUnitOfWork()
+
+    def because_we_report_a_new_issue(self):
+        handler = ReportIssueHandler(self.uow)
+        cmd = ReportIssueCommand(name, email, desc)
+
+        handler.handle(cmd)
+
+    def the_handler_should_have_created_a_new_issue(self):
+        expect(self.uow.issues).to(have_len(1))
+
+    def it_should_have_recorded_the_issuer(self):
+        expect(self.uow.issues[0].reporter.name).to(equal(name))
+        expect(self.uow.issues[0].reporter.email).to(equal(email))
+
+    def it_should_have_recorded_the_description(self):
+        expect(self.uow.issues[0].description).to(equal(desc))
+
+    def it_should_have_committed_the_unit_of_work(self):
+        expect(self.uow.was_committed).to(be_true)
+
+ +

Next time [https://io.made.com/blog/commands-and-queries-handlers-and-views] +we’ll look at how to get data back out of the system.

+
+ +
+ + + + + + + +
+
+ + \ No newline at end of file diff --git a/docs/blog/2017-09-13-commands-and-queries-handlers-and-views.html b/blog/2017-09-13-commands-and-queries-handlers-and-views.html similarity index 91% rename from docs/blog/2017-09-13-commands-and-queries-handlers-and-views.html rename to blog/2017-09-13-commands-and-queries-handlers-and-views.html index e58281f..47f427f 100644 --- a/docs/blog/2017-09-13-commands-and-queries-handlers-and-views.html +++ b/blog/2017-09-13-commands-and-queries-handlers-and-views.html @@ -2,25 +2,35 @@ - - - - + + + Commands, Handlers, Queries and Views + + + + + + + + + + + -
+
-
diff --git a/blog/2017-09-19-why-use-domain-events.html b/blog/2017-09-19-why-use-domain-events.html new file mode 100644 index 0000000..6936a45 --- /dev/null +++ b/blog/2017-09-19-why-use-domain-events.html @@ -0,0 +1,578 @@ + + + + + + + Why use domain events? + + + + + + + + + + + + + + + + + + + +
+ + + +
+ +

Why use domain events?

+

by Bob, 2017-09-19

+ + + + + +
+

Nota bene: this instalment in the Ports and Adapters with Command Handlers +series is code-heavy, and isn’t going to make much sense unless you’ve read the +previous parts:

+ +

Okay, so we have a basic skeleton for an application and we can add new issues +into the database, then fetch them from a Flask API. So far, though, we don’t +have any domain logic at all. All we have is a whole bunch of complicated crap +where we could just have a tiny Django app. Let’s work through some more +use-cases and start to flesh things out.

+

Back to our domain expert:

+

So when we’ve added a reported issue to the issue log, what happens next?

+
+

Well we need to triage the problem and decide how urgent it is. Then we might +assign it to a particular engineer, or we might leave it on the queue to be +picked up by anyone.

+
+

Wait, the queue? I thought you had an issue log, are they the same thing, or is +there a difference?

+
+

Oh, yes. The issue log is just a record of all the issues we have received, but +we work from the queue.

+
+

I see, and how do things get into the queue?

+
+

We triage the new items in the issue log to decide how urgent they are, and what +categories they should be in. When we know how to categorise them, and how +urgent they are, we treat the issues as a queue, and work through them in +priority order.

+
+

This is because users always set things to “Extremely urgent”?

+
+

Yeah, it’s just easier for us to triage the issues ourselves.

+
+

And what does that actually mean, like, do you just read the ticket and say “oh, +this is 5 important, and it’s in the broken mouse category”?

+
+

Mmmm… more or less, sometimes we need to ask more questions from the user so +we’ll email them, or call them. Most things are first-come, first-served, but +occasionally someone needs a fix before they can go to a meeting or something.

+
+

So you email the user to get more information, or you call them up, and then you +use that information to assess the priority of the issue - sorry triage the +issue, and work out what category it should go in… what do the categories +achieve? Why categorise?

+
+

Partly for reporting, so we can see what stuff is taking up the most time, or if +there are clusters of similar problems on a particular batch of laptops for +example. Mostly because different engineers have different skills, like if you +have a problem with the Active Directory domain, then you should send that to +Barry, or if it’s an Exchange problem, then George can sort it out, and Mike has +the equipment log so he can give you a temporary laptop and so on, and so on.

+
+

Okay, and where do I find this “queue”?

+
+

Your customer grins and gestures at the wall where a large whiteboard is covered +in post-its and stickers of different colours.

+
+

Mapping our requirements to our domain

+

How can we map these requirements back to our system? Looking back over our +notes with the domain expert, there’s a few obvious verbs that we should use to +model our use cases. We can triage an issue, which means we prioritise and +categorise it; we can assign a triaged issue to an engineer, or an engineer can + pick up an unassigned issue. There’s also a whole piece about asking +questions, which we might do synchronously by making a phone call and filling +out some more details, or asynchronously by sending an email. The Queue, with +all of its stickers and sigils and swimlanes looks too complicated to handle +today, so we’ll dig deeper into that separately.

+

Let’s quickly flesh out the triage use cases. We’ll start by updating the +existing unit test for reporting an issue:

+
class When_reporting_an_issue:
+
+    def given_an_empty_unit_of_work(self):
+        self.uow = FakeUnitOfWork()
+
+    def because_we_report_a_new_issue(self):
+        handler = ReportIssueHandler(self.uow)
+        cmd = ReportIssueCommand(id, name, email, desc)
+        handler.handle(cmd)
+
+    @property
+    def issue(self):
+        return self.uow.issues[0]
+
+    def it_should_be_awaiting_triage(self):
+        expect(self.issue.state).to(equal(IssueState.AwaitingTriage))
+
+ +

We’re introducing a new concept - Issues now have a state, and a newly reported +issue begins in the AwaitingTriage state. We can quickly add a command and +handler that allows us to triage an issue.

+
class TriageIssueHandler:
+
+    def __init__(self, uowm: UnitOfWorkManager):
+        self.uowm = uowm
+
+    def handle(self, cmd):
+        with self.uowm.start() as uow:
+            issue = uow.issues.get(cmd.issue_id)
+            issue.triage(cmd.priority, cmd.category)
+            uow.commit()
+
+ +

Triaging an issue, for now, is a matter of selecting a category and priority. +We’ll use a free string for category, and an enumeration for Priority. Once an +issue is triaged, it enters the AwaitingAssignment state. At some point we’ll +need to add some view builders to list issues that are waiting for triage or +assignment, but for now let’s quickly add a handler so that an engineer can Pick + an issue from the queue.

+
class PickIssueHandler:
+
+    def __init__(self, uowm: UnitOfWorkManager):
+        self.uowm = uowm
+
+    def handle(self, cmd):
+        with self.uowm.start() as uow:
+            issue = uow.issues.get(cmd.issue_id)
+            issue.assign_to(cmd.picked_by)
+            uow.commit()
+
+ +

At this point, the handlers are becoming a little boring. As I said way back in +the first part [https://io.made.com/blog/introducing-command-handler/], commands +handlers are supposed to be boring glue-code, and every command handler has the +same basic structure:

+
    +
  1. Fetch current state.
  2. +
  3. Mutate the state by calling a method on our domain model.
  4. +
  5. Persist the new state.
  6. +
  7. Notify other parts of the system that our state has changed.
  8. +
+

So far, though, we’ve only seen steps 1, 2, and 3. Let’s introduce a new +requirement.

+

When an issue is assigned to an engineer, can we send them an email to let them +know?

+

A brief discourse on SRP +Let’s try and implement this new requirement. Here’s a first attempt:

+
class AssignIssueHandler:
+
+    def __init__(self,
+               uowm: UnitOfWorkManager,
+               email_builder: EmailBuilder,
+               email_sender: EmailSender):
+        self.uowm = uowm
+        self.email_builder = email_builder
+        self.email_sender = email_sender
+
+    def handle(self, cmd):
+        # Assign Issue
+        with self.uowm.start() as uow:
+            issue = uow.issues.get(cmd.issue_id)
+            issue.assign_to(
+                cmd.assigned_to,
+                assigned_by=cmd.assigned_by
+            )
+            uow.commit()
+
+        # Send Email
+        email = self.email_builder.build(
+                cmd.assigned_to,
+                cmd.assigned_by,
+                issue.problem_description)
+        self.email_sender.send(email)
+
+ +

Something here feels wrong, right? Our command-handler now has two very distinct +responsibilities. Back at the beginning of this series we said we would stick +with three principles:

+
    +
  1. We will always define where our use-cases begin and end.
  2. +
  3. We will depend on abstractions, and not on concrete implementations.
  4. +
  5. We will treat glue code as distinct from business logic, and put it in an + appropriate place.
  6. +
+

The latter two are being maintained here, but the first principle feels a little +more strained. At the very least we’re violating the Single Responsibility +Principle [https://en.wikipedia.org/wiki/Single_responsibility_principle]; my +rule of thumb for the SRP is “describe the behaviour of your class. If you use +the word ‘and’ or ‘then’ you may be breaking the SRP”. What does this class do? +It assigns an issue to an engineer, AND THEN sends them an email. That’s enough +to get my refactoring senses tingling, but there’s another, less theoretical, +reason to split this method up, and it’s to do with error handling.

+

If I click a button marked “Assign to engineer”, and I can’t assign the issue to +that engineer, then I expect an error. The system can’t execute the command I’ve +given to it, so I should retry, or choose a different engineer.

+

If I click a button marked “Assign to engineer”, and the system succeeds, but +then can’t send a notification email, do I care? What action should I take in +response? Should I assign the issue again? Should I assign it to someone else? +What state will the system be in if I do?

+

Looking at the problem in this way, it’s clear that “assigning the issue” is the +real boundary of our use case, and we should either do that successfully, or +fail completely. “Send the email” is a secondary side effect. If that part fails +I don’t want to see an error - let the sysadmins clear it up later.

+

What if we split out the notification to another class?

+
class AssignIssueHandler:
+
+    def __init__(self, uowm: UnitOfWorkManager):
+        self.uowm = uowm
+
+    def handle(self, cmd):
+        with self.uowm.start() as uow:
+            issue = uow.issues.get(cmd.issue_id)
+            issue.assign_to(
+                cmd.assignee_address,
+                assigned_by=cmd.assigner_address
+            )
+            uow.commit()
+
+
+class SendAssignmentEmailHandler
+    def __init__(self,
+               uowm: UnitOfWorkManager,
+               email_builder: EmailBuilder,
+               email_sender: EmailSender):
+        self.uowm = uowm
+        self.email_builder = email_builder
+        self.email_sender = email_sender
+
+    def handle(self, cmd):
+        with self.uowm.start() as uow:
+            issue = uow.issues.get(cmd.issue_id)
+
+            email = self.email_builder.build(
+                cmd.assignee_address,
+                cmd.assigner_address,
+                issue.problem_description)
+            self.email_sender.send(email)
+
+ +

We don’t really need a unit of work here, because we’re not making any +persistent changes to the Issue state, so what if we use a view builder instead?

+
class SendAssignmentEmailHandler
+    def __init__(self,
+               view: IssueViewBuilder,
+               email_builder: EmailBuilder,
+               email_sender: EmailSender):
+        self.view = view
+        self.email_builder = email_builder
+        self.email_sender = email_sender
+
+    def handle(self, cmd):
+        issue = self.view.fetch(cmd.issue_id)
+
+        email = self.email_builder.build(
+            cmd.assignee_address,
+            cmd.assigner_address,
+            issue['problem_description'])
+        self.email_sender.send(email)
+
+ +

That seems better, but how should we invoke our new handler? Building a new +command and handler from inside our AssignIssueHandler also sounds like a +violation of SRP. Worse still, if we start calling handlers from handlers, we’ll +end up with our use cases coupled together again - and that’s definitely a +violation of Principle #1.

+

What we need is a way to signal between handlers - a way of saying “I did my +job, can you go do yours?”

+

All Aboard the Message Bus +In this kind of system, we use Domain Events +[http://verraes.net/2014/11/domain-events/] to fill that need. Events are +closely related to Commands, in that both commands and events are types of +message +[http://www.enterpriseintegrationpatterns.com/patterns/messaging/Message.html] +- named chunks of data sent between entities. Commands and events differ only in +their intent:

+
    +
  1. Commands are named with the imperative tense (Do this thing), events are + named in the past tense (Thing was done).
  2. +
  3. Commands must be handled by exactly one handler, events can be handled by 0 + to N handlers.
  4. +
  5. If an error occurs when processing a command, the entire request should + fail. If an error occurs while processing an event, we should fail + gracefully.
  6. +
+

We will often use domain events to signal that a command has been processed and +to do any additional book-keeping. When should we use a domain event? Going back +to our principle #1, we should use events to trigger workflows that fall outside +of our immediate use-case boundary. In this instance, our use-case boundary is +“assign the issue”, and there is a second requirement “notify the assignee” that +should happen as a secondary result. Notifications, to humans or other systems, +are one of the most common reasons to trigger events in this way, but they might +also be used to clear a cache, or regenerate a view model, or execute some logic +to make the system eventually consistent.

+

Armed with this knowledge, we know what to do - we need to raise a domain event +when we assign an issue to an engineer. We don’t want to know about the +subscribers to our event, though, or we’ll remain coupled; what we need is a +mediator, a piece of infrastructure that can route messages to the correct +places. What we need is a message bus. A message bus is a simple piece of +middleware that’s responsible for getting messages to the right listeners. In +our application we have two kinds of message, commands and events. These two +types of message are in some sense symmetrical, so we’ll use a single message +bus for both.

+

How do we start off writing a message bus? Well, it needs to look up subscribers +based on the name of an event. That sounds like a dict to me:

+
class MessageBus:
+
+    def __init__(self):
+        """Our message bus is just a mapping from message type
+           to a list of handlers"""
+        self.subscribers = defaultdict(list)
+
+    def handle(self, msg):
+        """The handle method invokes each handler in turn
+           with our event"""
+        msg_name = type(msg).__name__
+        subscribers = self.subscribers[msg_name]
+        for subscriber in subscribers:
+            subscriber.handle(cmd)
+
+    def subscribe_to(self, msg, handler):
+        """Subscribe sets up a new mapping, we make sure not
+           to allow more than one handler for a command"""
+        subscribers = [msg.__name__]
+        if msg.is_cmd and len(subscribers) > 0:
+           raise CommandAlreadySubscribedException(msg.__name__)
+        subscribers.append(handler)
+
+# Example usage
+bus = MessageBus()
+bus.subscribe_to(ReportIssueCommand, ReportIssueHandler(db.unit_of_work_manager))
+bus.handle(cmd)
+
+ +

Here we have a bare-bones implementation of a message bus. It doesn’t do +anything fancy, but it will do the job for now. In a production system, the +message bus is an excellent place to put cross-cutting concerns; for example, we +might want to validate our commands before passing them to handlers, or we may +want to perform some basic logging, or performance monitoring. I want to talk +more about that in the next part, when we’ll tackle the controversial subject of +dependency injection and Inversion of Control containers.

+

For now, let’s look at how to hook this up. Firstly, we want to use it from our +API handlers.

+
@api.route('/issues', methods=['POST'])
+def create_issue(self):
+    issue_id = uuid.uuid4()
+    cmd = ReportIssueCommand(issue_id=issue_id, **request.get_json())
+    bus.handle(cmd)
+    return "", 201, {"Location": "/issues/" + str(issue_id) }
+
+ +

Not much has changed here - we’re still building our command in the Flask +adapter, but now we’re passing it into a bus instead of directly constructing a +handler for ourselves. What about when we need to raise an event? We’ve got +several options for doing this. Usually I raise events from my command handlers, +like this:

+
class AssignIssueHandler:
+
+    def handle(self, cmd):
+        with self.uowm.start() as uow:
+            issue = uow.issues.get(cmd.id)
+            issue.assign_to(cmd.assigned_to, cmd.assigned_by)
+            uow.commit()
+
+        # This is step 4: notify other parts of the system
+        self.bus.raise(IssueAssignedToEngineer(
+            cmd.issue_id,
+            cmd.assigned_to,
+            cmd.assigned_by))
+
+ +

I usually think of this event-raising as a kind of glue - it’s orchestration +code. Raising events from your handlers this way makes the flow of messages +explicit - you don’t have to look anywhere else in the system to understand +which events will flow from a command. It’s also very simple in terms of +plumbing. The counter argument is that this feels like we’re violating SRP in +exactly the same way as before - we’re sending a notification about our +workflow. Is this really any different to sending the email directly from the +handler? Another option is to send events directly from our model objects, and +treat them as part our domain model proper.

+
class Issue:
+
+    def assign_to(self, assigned_to, assigned_by):
+        self.assigned_to = assigned_to
+        self.assigned_by = assigned_by
+
+        # Add our new event to a list
+        self.events.add(IssueAssignedToEngineer(self.id, self.assigned_to, self.assigned_by))
+
+ +

There’s a couple of benefits of doing this: firstly, it keeps our command +handler simpler, but secondly it pushes the logic for deciding when to send an +event into the model. For example, maybe we don’t always need to raise the +event.

+
class Issue:
+
+    def assign_to(self, assigned_to, assigned_by):
+        self.assigned_to = assigned_to
+        self.assigned_by = assigned_by
+
+        # don't raise the event if I picked the issue myself
+        if self.assigned_to != self.assigned_by:
+            self.events.add(IssueAssignedToEngineer(self.id, self.assigned_to, self.assigned_by))
+
+ +

Now we’ll only raise our event if the issue was assigned by another engineer. +Cases like this are more like business logic than glue code, so today I’m +choosing to put them in my domain model. Updating our unit tests is trivial, +because we’re just exposing the events as a list on our model objects:

+
class When_assigning_an_issue:
+
+    issue_id = uuid.uuid4()
+    assigned_to = 'ashley@example.org'
+    assigned_by = 'laura@example.org'
+
+    def given_a_new_issue(self):
+        self.issue = Issue(self.issue_id, 'reporter@example.org', 'how do I even?')
+
+    def because_we_assign_the_issue(self):
+        self.issue.assign(self.assigned_to, self.assigned_by)
+
+    def we_should_raise_issue_assigned(self):
+        expect(self.issue).to(have_raised(
+            IssueAssignedToEngineer(self.issue_id,
+                                    self.assigned_to,
+                                    self.assigned_by)))
+
+ +

The have_raised function is a custom matcher I wrote that checks the events +attribute of our object to see if we raised the correct event. It’s easy to test +for the presence of events, because they’re namedtuples, and have value +equality.

+

All that remains is to get the events off our model objects and into our message +bus. What we need is a way to detect that we’ve finished one use-case and are +ready to flush our changes. Fortunately, we have a name for this already - it’s +a unit of work. In this system I’m using SQLAlchemy’s event hooks +[http://docs.sqlalchemy.org/en/latest/orm/session_events.html] to work out +which objects have changed, and queue up their events. When the unit of work +exits, we raise the events.

+
class SqlAlchemyUnitOfWork(UnitOfWork):
+
+    def __init__(self, sessionfactory, bus):
+        self.sessionfactory = sessionfactory
+        self.bus = bus
+        # We want to listen to flush events so that we can get events
+        # from our model objects
+        event.listen(self.sessionfactory, "after_flush", self.gather_events)
+
+    def __enter__(self):
+        self.session = self.sessionfactory()
+        # When we first start a unit of work, create a list of events
+        self.flushed_events = []
+        return self
+
+    def commit(self):
+        self.session.flush()
+        self.session.commit()
+
+    def rollback(self):
+        self.session.rollback()
+        # If we roll back our changes we should drop all the events
+        self.events = []
+
+    def gather_events(self, session, ctx):
+        # When we flush changes, add all the events from our new and
+        # updated entities into the events list
+        flushed_objects = ([e for e in session.new]
+                        + [e for e in session.dirty])
+        for e in flushed_objects:
+            self.flushed_events += e.events
+
+    def publish_events(self):
+        # When the unit of work completes
+        # raise any events that are in the list
+        for e in self.flushed_events:
+            self.bus.handle(e)
+
+    def __exit__(self, type, value, traceback):
+        self.session.close()
+        self.publish_events()
+
+ +

Okay, we’ve covered a lot of ground here. We’ve discussed why you might want to +use domain events, how a message bus actually works in practice, and how we can +get events out of our domain and into our subscribers. The newest code sample +[https://github.com/bobthemighty/blog-code-samples/tree/master/ports-and-adapters/04] + demonstrates these ideas, please do check it out, run it, open pull requests, +open Github issues etc.

+

Some people get nervous about the design of the message bus, or the unit of +work, but this is just infrastructure - it can be ugly, so long as it works. +We’re unlikely to ever change this code after the first few user-stories. It’s +okay to have some crufty code here, so long as it’s in our glue layers, safely +away from our domain model. Remember, we’re doing all of this so that our domain +model can stay pure and be flexible when we need to refactor. Not all layers of +the system are equal, glue code is just glue.

+

Next time I want to talk about Dependency Injection, why it’s great, and why +it’s nothing to be afraid of.

+
+ +
+ + + + + + + +
+
+ + \ No newline at end of file diff --git a/blog/2019-04-15-inversion-of-control.html b/blog/2019-04-15-inversion-of-control.html new file mode 100644 index 0000000..cba8af9 --- /dev/null +++ b/blog/2019-04-15-inversion-of-control.html @@ -0,0 +1,186 @@ + + + + + + + What is Inversion of Control and Why Does it Matter? + + + + + + + + + + + + + + + + + + + +
+ + + +
+ +

What is Inversion of Control and Why Does it Matter?

+

by David, 2019-04-15

+ + +
+
+ + +
+
+ + +
+

David was a tech reviewer for the book and these two excellent +articles on inversion of control are cross-posted from +his blog where you can find lots more excellent content.

+

When I first learned to program, the code I wrote all followed a particular pattern: I wrote instructions to the computer +that it would execute, one by one. If I wanted to make use of utilities written elsewhere, such as in a third party library, +I would call those utilities directly from my code. Code like this could be described as employing the ‘traditional flow of control’. +Perhaps it’s just my bias, but this still seems to me to be the obvious way to program.

+

Despite this, there is a wider context that the majority of the code I write today runs in; a context where control is being inverted. +This is because I’m usually using some kind of framework, which is passing control to my code, despite having no direct dependency on it. +Rather than my code calling the more generic code, the framework allows me to plug in custom behaviour. +Systems designed like this are using what is known as Inversion of Control +(IoC for short).

+

This situation can be depicted like so: the generic framework providing points where the custom code can insert its behaviour.

+

Framework with custom behaviours plugged in

+

Even though many of us are familiar with coding in the context of such a framework, we tend to be reticent to apply the +same ideas in the software that we design. Indeed, it may seem a bizarre or even impossible thing to do. It is certainly +not the ‘obvious’ way to program.

+

But IoC need not be limited to frameworks — on the contrary, it is a particularly useful tool in a programmer’s belt. +For more complex systems, it’s one of the best ways to avoid our code getting into a mess. Let me tell you why.

+

Striving for modularity

+

Software gets complicated easily. Every programmer has experienced tangled, difficult-to-work with code. +Here’s a diagram of such a system:

+

A single complicated system

+

Perhaps not such a helpful diagram, but some systems can feel like this to work with: a forbidding mass +of code that feels impossible to wrap one’s head around.

+

A common approach to tackling such complexity is to break up the system into smaller, more manageable parts. +By separating it into simpler subsystems, the aim is to reduce complexity and allow us to think more clearly +about each one in turn.

+

A system composed of small simple modules

+

We call this quality of a system its modularity, and we can refer to these subsystems as modules.

+

Separation of concerns

+

Most of us recognise the value of modularity, and put effort into organising our code into smaller parts. We have to +decide what goes into which part, and the way we do this is by the separation of concerns.

+

This separation can take different forms. We might organize things by feature area +(the authentication system, the shopping cart, the blog) or by level of detail +(the user interface, the business logic, the database), or both.

+

When we do this, we tend to be aiming at modularity. Except for some reason, the system remains complicated. +In practice, working on one module needs to ask questions of another part of the system, +which calls another, which calls back to the original one. Soon our heads hurt and we need to have +a lie down. What’s going wrong?

+

Separation of concerns is not enough

+

The sad fact is, if the only organizing factor of code is separation of concerns, a system will not be +modular after all. Instead, separate parts will tangle together.

+

Pretty quickly, our efforts to organise what goes into each module are undermined by the relationships between those +modules.

+

This is naturally what happens to software if you don’t think about relationships. This is because in the real world +things are a messy, interconnected web. As we build functionality, we realise that one module needs to know about +another. Later on, that other module needs to know about the first. Soon, everything knows about everything else.

+

A complicated system with lots of arrows between the modules

+

The problem with software like this is that, because of the web of relationships, it is not a collection of smaller +subsystems. Instead, it is a single, large system - and large systems tend to be more complicated than smaller ones.

+

Improving modularity through decoupling

+

The crucial problem here is that the modules, while appearing separate, are tightly coupled by their dependencies +upon one other. Let’s take two modules as an example:

+

Arrows pointing in both directions between A and B

+

In this diagram we see that A depends on B, but B also depends upon A. It’s a +circular dependency. As a result, these two modules are in fact no less complicated than a single module. +How can we improve things?

+

Removing cycles by inverting control

+

There are a few ways to tackle a circular dependency. You may be able to extract a shared dependency into a separate +module, that the other two modules depend on. You may be able to create an extra module that coordinates the two modules, +instead of them calling each other. Or you can use inversion of control.

+

At the moment, each module calls each other. We can pick one of the calls (let’s say A’s call to B) and invert +control so that A no longer needs to know anything about B. Instead, it exposes a way of plugging into its +behaviour, that B can then exploit. This can be diagrammed like so:

+

B plugging into A

+

Now that A has no specific knowledge of B, we think about A in isolation. We’ve just reduced our mental overhead, +and made the system more modular.

+

The tactic remains useful for larger groups of modules. For example, three modules may depend upon each other, in +a cycle:

+

Arrows pointing from A to B to C, and back to A

+

In this case, we can invert one of the dependencies, gaining us a single direction of flow:

+

B plugging into A

+

Again, inversion of control has come to the rescue.

+

Inversion of control in practice

+

In practice, inverting control can sometimes feel impossible. Surely, if a module needs to call another, there is no way +to reverse this merely by refactoring? But I have good news. You should always be able to avoid circular dependencies +through some form of inversion (if you think you’ve found an example where it isn’t, please tell me). +It’s not always the most obvious way to write code, but it can make your code base significantly easier to work with.

+

There are several different techniques for how you do this. One such technique that is often + talked about is dependency injection. I will cover some of these techniques in part two of this series.

+

There is also more to be said about how to apply this approach across the wider code base: if the system consists of +more than a handful of files, where do we start? Again, I’ll cover this later in the series.

+

Conclusion: complex is better than complicated

+

If you want to avoid your code getting into a mess, it’s not enough merely to separate concerns. You must control the +relationships between those concerns. In order to gain the benefits of a more modular system, you will sometimes need +to use inversion of control to make control flow in the opposite direction to what comes naturally.

+

The Zen of Python states:

+
Simple is better than complex.
+
+ +

But also that

+
Complex is better than complicated.
+
+ +

I think of inversion of control as an example of choosing the complex over the complicated. If we don’t use it when +it’s needed, our efforts to create a simple system will tangle into complications. Inverting dependencies allows us, +at the cost of a small amount of complexity, to make our systems less complicated.

+

Further information

+ +
+ +
+ + + + + + + +
+
+ + \ No newline at end of file diff --git a/blog/2019-08-03-ioc-techniques.html b/blog/2019-08-03-ioc-techniques.html new file mode 100644 index 0000000..9a30102 --- /dev/null +++ b/blog/2019-08-03-ioc-techniques.html @@ -0,0 +1,413 @@ + + + + + + + Three Techniques for Inverting Control, in Python + + + + + + + + + + + + + + + + + + + +
+ + + +
+ +

Three Techniques for Inverting Control, in Python

+

by David, 2019-08-03

+ + +
+
+ + +
+
+ + +
+

David was a tech reviewer for the book and these two excellent +articles on inversion of control are cross-posted from +his blog where you can find lots more excellent content.

+

In the previous post we learned how Inversion of Control can +be visualised as follows:

+

B plugging into A

+

B plugs into A. A provides a mechanism for B to do this — but otherwise A need know nothing about B.

+

The diagram provides a high level view of the mechanism, but how is this actually implemented?

+

A pattern for inverting control

+

Getting a little closer to the code structure, we can use this powerful pattern:

+

main pointing to A and B, A pointing to <B>, B pointing (open arrow) to <B>

+

This is the basic shape of inversion of control. Captured within the notation, which may or may not be familiar +to you, are the concepts of abstraction, implementation and interface. These concepts are all important +to understanding the techniques we’ll be employing. Let’s make sure we understand what they mean when applied +to Python.

+

Abstractions, implementations and interfaces — in Python

+

Consider three Python classes:

+
class Animal:
+    def speak(self):
+        raise NotImplementedError
+
+
+class Cat(Animal):
+    def speak(self):
+        print("Meow.")
+
+
+class Dog(Animal):
+    def speak(self):
+        print("Woof.")
+
+ +

In this example, Animal is an abstraction: it declares its speak method, but it’s not intended to be run (as +is signalled by the NotImplementedError).

+

Cat and Dog, however, are implementations: they both implement the speak method, each in their own way.

+

The speak method can be thought of as an interface: a common way in which other code may interact with +these classes.

+

This relationship of classes is often drawn like this, with an open arrow indicating that Cat and Dog are concrete +implementations of Animal.

+

Diagram of Cat and Dog subclassing Animal

+

Polymorphism and duck typing

+

Because Cat and Dog implement a shared interface, we can interact with either class without knowing which one it is:

+
def make_animal_speak(animal):
+    animal.speak()
+
+
+make_animal_speak(Cat())
+make_animal_speak(Dog())
+
+ +

The make_animal_speak function need not know anything about cats or dogs; all it has to know is how to interact +with the abstract concept of an animal. Interacting with objects without knowing +their specific type, only their interface, is known as ‘polymorphism’.

+

Of course, in Python we don’t actually need the base class:

+
class Cat:
+    def speak(self):
+        print("Meow.")
+
+
+class Dog:
+    def speak(self):
+        print("Woof.")
+
+ +

Even if Cat and Dog don’t inherit Animal, they can still be passed to make_animal_speak and things +will work just fine. This informal ability to interact with an object without it explicitly declaring an interface +is known as ‘duck typing’.

+

We aren’t limited to classes; functions may also be used in this way:

+
def notify_by_email(customer, event):
+    ...
+
+
+def notify_by_text_message(customer, event):
+    ...
+
+
+for notify in (notify_by_email, notify_by_text_message):
+    notify(customer, event)
+
+ +

We may even use Python modules:

+
import email
+import text_message
+
+
+for notification_method in (email, text_message):
+    notification_method.notify(customer, event)
+
+ +

Whether a shared interface is manifested in a formal, object oriented manner, or more implicitly, we can +generalise the separation between the interface and the implementation like so:

+

Diagram of implementation inheriting abstract interface

+

This separation will give us a lot of power, as we’ll see now.

+

A second look at the pattern

+

Let’s look again at the Inversion of Control pattern.

+

main pointing to A and B, A pointing to <B>, B pointing (open arrow) to <B>

+

In order to invert control between A and B, we’ve added two things to our design.

+

The first is <<B>>. We’ve separated out into its abstraction (which A will continue to depend on and know about), +from its implementation (of which A is blissfully ignorant).

+

However, somehow the software will need to make sure that B is used in place of its abstraction. We therefore need +some orchestration code that knows about both A and B, and does the final linking of them together. I’ve called +this main.

+

It’s now time to look at the techniques we may use for doing this.

+

Technique One: Dependency Injection

+

Dependency Injection is where a piece of code allows the calling code to control its dependencies.

+

Let’s begin with the following function, which doesn’t yet support dependency injection:

+
# hello_world.py
+
+
+def hello_world():
+    print("Hello, world.")
+
+ +

This function is called from a top level function like so:

+
# main.py
+
+from hello_world import hello_world
+
+
+if __name__ == "__main__":
+    hello_world()
+
+ +

hello_world has one dependency that is of interest to us: the built in function print. We can draw a diagram +of these dependencies like this:

+

Main pointing to hello_world pointing to print

+

The first step is to identify the abstraction that print implements. We could think of this simply as a +function that outputs a message it is supplied — let’s call it output_function.

+

Now, we adjust hello_world so it supports the injection of the implementation of output_function. Drum roll please…

+
# hello_world.py
+
+
+def hello_world(output_function):
+    output_function("Hello, world.")
+
+ +

All we do is allow it to receive the output function as an argument. The orchestration code then passes in the print function via the argument:

+
# main.py
+
+import hello_world
+
+
+if __name__ == "__main__":
+    hello_world.hello_world(output_function=print)
+
+ +

That’s it. It couldn’t get much simpler, could it? In this example, we’re injecting a callable, but other +implementations could expect a class, an instance or even a module.

+

With very little code, we have moved the dependency out of hello_world, into the top level function:

+

Main pointing to hello_world and print, hello_world pointing to <output>, print pointing (open arrow) to <output>.

+

Notice that although there isn’t a formally declared abstract output_function, that concept is implicitly there, so +I’ve included it in the diagram.

+

Technique Two: Registry

+

A Registry is a store that one piece of code reads from to decide how to behave, which may be +written to by other parts of the system. Registries require a bit more machinery that dependency injection.

+

They take two forms: Configuration and Subscriber:

+

The Configuration Registry

+

A configuration registry gets populated once, and only once. A piece of code uses one +to allow its behaviour to be configured from outside.

+

Although this needs more machinery than dependency injection, it doesn’t need much:

+
# hello_world.py
+
+
+config = {}
+
+
+def hello_world():
+    output_function = config["OUTPUT_FUNCTION"]
+    output_function("Hello, world.")
+
+ +

To complete the picture, here’s how it could be configured externally:

+
# main.py
+
+import hello_world
+
+
+hello_world.config["OUTPUT_FUNCTION"] = print
+
+
+if __name__ == "__main__":
+    hello_world.hello_world()
+
+ +

The machinery in this case is simply a dictionary that is written to from outside the module. In a real world system, +we might want a slightly more sophisticated config system (making it immutable for example, is a good idea). But at heart, +any key-value store will do.

+

As with dependency injection, the output function’s implementation has been lifted out, so hello_world no longer depends on it.

+

Configuration registry

+

The Subscriber Registry

+

In contrast to a configuration registry, which should only be populated once, a +subscriber registry may be populated an arbitrary number of times by different parts +of the system.

+

Let’s develop our ultra-trivial example to use this pattern. Instead of saying “Hello, world”, we want +to greet an arbitrary number of people: “Hello, John.”, “Hello, Martha.”, etc. Other parts of the system should be +able to add people to the list of those we should greet.

+
# hello_people.py
+
+people = []
+
+
+def hello_people():
+    for person in people:
+        print(f"Hello, {person}.")
+
+ +
# john.py
+
+import hello_people
+
+
+hello_people.people.append("John")
+
+ +
# martha.py
+
+import hello_people
+
+
+hello_people.people.append("Martha")
+
+ +

As with the configuration registry, there is a store that can be written to from outside. But instead of +being a dictionary, it’s a list. This list is populated, typically +at startup, by other components scattered throughout the system. When the time is right, +the code works through each item one by one.

+

A diagram of this system would be:

+

Subscriber registry

+

Notice that in this case, main doesn’t need to know about the registry — instead, it’s the subscribers elsewhere +in the system that write to it.

+

Subscribing to events

+

A common reason for using a subscriber registry is to allow other parts of a system to react to events +that happen one place, without that place directly calling them. This is often solved by the Observer Pattern, +a.k.a. pub/sub.

+

We may implement this in much the same way as above, except instead of adding strings to a list, we add callables:

+
# hello_world.py
+
+subscribers = []
+
+
+def hello_world():
+    print("Hello, world.")
+    for subscriber in subscribers:
+        subscriber()
+
+ +
# log.py
+
+import hello_world
+
+
+def write_to_log():
+    ...
+
+
+hello_world.subscribers.append(write_to_log)
+
+ +

Technique Three: Monkey Patching

+

Our final technique, Monkey Patching, is very different to the others, as it doesn’t use the Inversion of Control +pattern described above.

+

If our hello_world function doesn’t implement any hooks for injecting its output function, we could monkey patch the +built in print function with something different:

+
# main.py
+
+import hello_world
+from print_twice import print_twice
+
+
+hello_world.print = print_twice
+
+
+if __name__ == "__main__":
+    hello_world.hello_world()
+
+ +

Monkey patching takes other forms. You could manipulate to your heart’s content some hapless class defined elsewhere +— changing attributes, swapping in other methods, and generally doing whatever you like to it.

+

Choosing a technique

+

Given these three techniques, which should you choose, and when?

+

When to use monkey patching

+

Code that abuses the Python’s dynamic power can be extremely +difficult to understand or maintain. The problem is that if you are reading monkey patched code, you have no clue +to tell you that it is being manipulated elsewhere.

+

Monkey patching should be reserved for desperate times, where you don’t have the ability to change the code you’re +patching, and it’s really, truly impractical to do anything else.

+

Instead of monkey patching, it’s much better to use one of the other inversion of control techniques. +These expose an API that formally provides the hooks that other code can use to change behaviour, which is easier +to reason about and predict.

+

A legitimate exception is testing, where you can make use of unittest.mock.patch. This is monkey patching, but it’s +a pragmatic way to manipulate dependencies when testing code. Even then, some people view testing like this as +a code smell.

+

When to use dependency injection

+

If your dependencies change at runtime, you’ll need dependency injection. Its alternative, the registry, +is best kept immutable. You don’t want to be changing what’s in a registry, except at application start up.

+

json.dumps is a good example from the standard library which uses +dependency injection. It serializes a Python object to a JSON string, but if the default encoding doesn’t support what +you’re trying to serialize, it allows you to pass in a custom encoder class.

+

Even if you don’t need dependencies to change, dependency injection is a good technique if you want a really simple way +of overriding dependencies, and don’t want the extra machinery of configuration.

+

However, if you are having to inject the same dependency a lot, you might find your code becomes rather unwieldy and +repetitive. This can also happen if you only need the dependency quite deep in the call stack, and are having to pass +it around a lot of functions.

+

When to use registries

+

Registries are a good choice if the dependency can be fixed at start up time. While you could use dependency injection +instead, the registry is a good way to keep configuration separate from the control flow code.

+

Use a configuration registry when you need something configured to a single value. If there is already a +configuration system in place (e.g. if you’re using a framework that has a way of providing global configuration) then +there’s even less extra machinery to set up. A good example of this is Django’s ORM, which provides a Python API around different database engines. The ORM does not depend on any one database engine; instead, +you configure your project to use a particular database engine +via Django’s configuration system.

+

Use a subscriber registry for pub/sub, or when you depend on an arbitrary number of values. Django signals, +which are a pub/sub mechanism, use this pattern. A rather different use case, also from Django, +is its admin site. This uses a subscriber registry to +allow different database tables to be registered with it, exposing a CRUD interface in the UI.

+

Configuration registries may be used in place of subscriber registries for configuring, +say, a list — if you prefer doing your linking up in single place, rather than scattering it throughout the application.

+

Conclusion

+

I hope these examples, which were as simple as I could think of, have shown how easy it is to invert control in Python. +While it’s not always the most obvious way to structure things, it can be achieved with very little extra code.

+

In the real world, you may prefer to employ these techniques with a bit more structure. I often choose classes rather +than functions as the swappable dependencies, as they allow you to declare the interface in a more formal way. +Dependency injection, too, has more sophisticated implementations, and there are even some third party frameworks available.

+

There are costs as well as benefits. Locally, code that employs IoC may be harder to understand and debug, so be sure that it +is reducing complication overall.

+

Whichever approaches you take, the important thing to remember is that the relationship of dependencies in a software package is +crucial to how easy it will be to understand and change. Following the path of least resistance can result in dependencies +being structured in ways that are, in fact, unnecessarily difficult to work with. These techniques give you the power +to invert dependencies where appropriate, allowing you to create more maintainable, modular code. Use them wisely!

+
+ +
+ + + + + + + +
+
+ + \ No newline at end of file diff --git a/docs/blog/2020-01-25-testing_external_api_calls.html b/blog/2020-01-25-testing_external_api_calls.html similarity index 96% rename from docs/blog/2020-01-25-testing_external_api_calls.html rename to blog/2020-01-25-testing_external_api_calls.html index 37da4b3..b14af5e 100644 --- a/docs/blog/2020-01-25-testing_external_api_calls.html +++ b/blog/2020-01-25-testing_external_api_calls.html @@ -2,25 +2,35 @@ - - - - + + + Writing tests for external API calls + + + + + + + + + + + -
+
-
diff --git a/blog/2020-05-12-ddia-review.html b/blog/2020-05-12-ddia-review.html new file mode 100644 index 0000000..1a7f16b --- /dev/null +++ b/blog/2020-05-12-ddia-review.html @@ -0,0 +1,119 @@ + + + + + + + Book review: Designing Data-Intensive Applications, by Martin Kleppmann + + + + + + + + + + + + + + + + + + + +
+ + + +
+ +

Book review: Designing Data-Intensive Applications, by Martin Kleppmann

+

by Harry, 2020-05-12

+ + + + + +
+

I bought this book on the strength of its reviews, and I am very happy to add +my own to the long list of five-stars.

+

The book’s aim is to help people to choose the right database technology for +their problem. It does so by explaining, at quite a decent level of detail:

+
    +
  • How various databases, algorithms and data structures work.
  • +
  • What guarantees they can give and what they cannot.
  • +
  • Examples of edge cases and unexpected behaviours.
  • +
+

It certainly has the potential to be unbelievably dry. But there are two +things, I think, that mean it actually turns out to be quite a page-turner The +first is if the reader is actually interested in the subject matter. +I certainly was; it filled a lot of gaps in my knowledge. But the second is +the quality of the writing. Somehow Kleppmann manages to give the whole thing +the feeling of the old “but wait, there’s more!” +comedy sketch trope.

+

Each chapter starts with a problem (eg “how do we manage concurrent access to +the database?” and then presents some seemingly straightforward solutions +(“transactions!”), then it goes on to explain them in detail, and along the way +we learn all sorts of horrible gotchas, and new, thornier, more subtle problems +that have been thrown up. And that leads us on to the next chapter, like an +unbelievably nerdy cliffhanger. I couldn’t put it down.

+

It’s really well balanced between academic and practical engineering concerns, +it’s extensively footnoted, it’s really well explained with good examples, and +it ends on a thoughtful, philosophical note. If any of this sounds appealing +at all, go read it no!

+

Plus he’s quite a fan of the event-driven approach ;-)

+ +
+ +
+ + + + + + + +
+
+ + \ No newline at end of file diff --git a/blog/2020-08-13-so-many-layers.html b/blog/2020-08-13-so-many-layers.html new file mode 100644 index 0000000..a49e5a3 --- /dev/null +++ b/blog/2020-08-13-so-many-layers.html @@ -0,0 +1,179 @@ + + + + + + + So Many Layers! A Note of Caution. + + + + + + + + + + + + + + + + + + + +
+ + + +
+ +

So Many Layers! A Note of Caution.

+

by Harry, 2020-08-13

+ + + + + +
+

In the book we are at pains to point out that each pattern is a trade-off, and +comes with costs. But just on the offchance that anyone was still missing the +message and thinking we were saying that all apps should be built like this, +I thought I’d write a small blog post just to reinforce the message about +costs. If you’ve been feeling tempted to cargo-cult every single pattern into +every single app from now on, this should put you off.

+

Each time you add a layer, you buy yourself some decoupling, but it +comes at the cost of an extra moving part. In the simplest terms, there’s an +extra file you have to maintain.

+

+
+ a recap of all the layers + parts of our architecture +
Here's a recap of all the layers + parts of our architecture
+
+

+

So. Once upon a time, early in my time at MADE, I remember having to make a +simple change to an app that the buying team uses. We needed to record an extra +piece of information for each shipment, an optional “delay” field to be used in +some ETA calculations. This is a nice illustration of a trip all the way +through the stack, because things have to change all the way from the +frontend/UI, all the way down to the database.

+

If you’re using a framework like Django, you might be used to thinking of a +change like this, in a perfect world, as a change you can make to just one file. +You would change models.py, and then your ModelForm will be updated automatically, +and maybe even the frontend will “just work” too, if you’re using the form’s +autogenerated HTML. That’s one of the reasons that Django is so good as a +rapid application development framework: by closely coupling its various parts, +it saves you a lot of messing about with database tables, html forms, +validation, and so on. And if those are the main things you spend your time on, +then Django is going to save you a lot of time.

+

But in our world (at least in theory *), +database tables and html forms are not where we spend our time. Instead, we +want to optimise for capturing and understand business logic, and as a +result we want to decouple things.

+

What does it cost? Well, let’s take a trip through each file I had to touch, +when I was making my very minor change to the data model in our app.

+
  1. + I started off with editing a selenium test of the frontend, plus a javascript + frontend test, plus the frontend javascript itself, plus an html template. + That's four files already, but they're not strictly relevant to the patterns + and layers whose cost I want to account for, so I'm going to say they don't + count. If you think I'm cheating, don't worry; there's plenty more to come. +
+ +

So:

+
    +
  1. An end-to-end / API test for the create and edit use cases for the objects in question.
  2. +
  3. The Command classes that capture those write interactions a user can have + with this model.
  4. +
  5. The Command schema which we use to validate incoming requests.
  6. +
  7. The Service-Layer tests which instantiates those commands to test their + handlers
  8. +
  9. The Handlers at the Service Layer that orchestrate these use cases.
  10. +
  11. The Domain Model tests that were affected. Although + not every domain model needs low-level unit tests as well as service-layer + tests, + so if I was being indulgent I might not count this. But we did happen to + have a few low-level tests in this case.
  12. +
  13. The Domain Model itself.
  14. +
  15. The Repository integration test + (repo and DB stuff is in chapter 3)
  16. +
  17. The Repository and ORM config
  18. +
  19. The database schema
  20. +
  21. A migration file (admittedly autogenerated by Alembic, but we like to just give them a bit of a tidy-up before committing).
  22. +
  23. The Event classes that capture ongoing internal / external consequences + of the various affected use cases
  24. +
  25. The Event schema files we use for (outbound) validation.
  26. +
  27. And that’s not all! Because this app uses CQRS, the read-side is separate from the +write side, so I also had to change some API JSON view tests
  28. +
  29. And the CQRS JSON views code
  30. +
+

So that’s fifteen files. Fifteen! To add one field!

+

Now I should add that each change was very simple. Most were a matter of +copy-pasting a line and some find+replace. The whole job might have taken an hour +or so. But if you’re used to this sort of thing taking five minutes and happening +in a single file, or at most a couple, then when first confronted with all these +layers, you are definitely going to start questioning the sanity of the entire +endeavour. I know I certainly did.

+

We think the cost we impose on ourselves here is worth it, because we believe +that the main thing we want to make easy is not adding database fields and html +forms. We want to make it easy to capture complex and evolving business +requirements in a domain model. But, as we try to say in each chapter, +your mileage may vary!

+
+ OK, in theory. In practice, I think this particular app was a _little_ + overengineered. It was one of the first ones that the team had complete + freedom to try new patterns on, and they may have gone to town a bit... + But on the other hand, there is now talk of converting that app to eventsourcing, + and thanks to all the layers, that would be relatively easy. Relatively. +
+
+ +
+ + + + + + + +
+
+ + \ No newline at end of file diff --git a/blog/2020-10-27-i-hate-enums.html b/blog/2020-10-27-i-hate-enums.html new file mode 100644 index 0000000..0097d44 --- /dev/null +++ b/blog/2020-10-27-i-hate-enums.html @@ -0,0 +1,214 @@ + + + + + + + Making Enums (as always, arguably) more Pythonic + + + + + + + + + + + + + + + + + + + +
+ + + +
+ +

Making Enums (as always, arguably) more Pythonic

+

by Harry, 2020-10-27

+ + + + + +
+

OK this isn’t really anything to do with software architecture, but:

+
+

I hate enums!

+
+

I thought to myself, again and again, when having to deal with them recently.

+

Why?

+
class BRAIN(Enum):
+    SMALL = 'small'
+    MEDIUM = 'medium'
+    GALAXY = 'galaxy'
+
+ +

What could be wrong with that, I hear you ask? +Well, accuse me of wanting to stringly type everything if you will, +but: those enums may look like strings but they aren’t!

+
assert BRAIN.SMALL == 'small'
+# nope, <BRAIN.SMALL: 'small'> != 'small'
+
+assert str(BRAIN.SMALL) == 'small'
+# nope, 'BRAIN.SMALL' != 'small'
+
+assert BRAIN.SMALL.value == 'small'
+# finally, yes.
+
+ +

I imagine some people think this is a feature rather than a bug? But for me +it’s an endless source of annoyance. They look like strings! I defined them +as strings! Why don’t they behave like strings arg!

+

Just one common motivating example: often what you want to do with those +enums is dump them into a database column somewhere. This not-quite-a-string +behaviour will cause your ORM or db-api library to complain like mad, and +no end of footguns and headscratching when writing tests, custom SQL, and so on. +At this point I’m wanting to throw them out and just use normal constants!

+

But, one of the nice promises from Python’s enum module is that it’s iterable. +So it’s easy not just to refer to one constant, +but also to refer to the list of all allowed constants. Maybe that’s enough +to want to rescue it?

+

But, again, it doesn’t quite work the way you might want it to:

+
assert list(BRAIN) == ['small', 'medium', 'galaxy']  # nope
+assert [thing for thing in BRAIN] == ['small', 'medium', 'galaxy']  # nope
+assert [thing.value for thing in BRAIN] == ['small', 'medium', 'galaxy']  # yes
+
+ +

Here’s a truly wtf one:

+
assert random.choice(BRAIN) in ['small', 'medium', 'galaxy']
+# Raises an Exception!!!
+
+  File "/usr/local/lib/python3.9/random.py", line 346, in choice
+    return seq[self._randbelow(len(seq))]
+  File "/usr/local/lib/python3.9/enum.py", line 355, in __getitem__
+    return cls._member_map_[name]
+KeyError: 2
+
+ +

I have no idea what’s going on there. What we actually wanted was

+
assert random.choice(list(BRAIN)) in ['small', 'medium', 'galaxy']
+# which is still not true, but at least it doesn't raise an exception
+
+ +

Now the standard library does provide a solution +if you want to duck-type your enums to integers, +IntEnum

+
class IBRAIN(IntEnum):
+    SMALL = 1
+    MEDIUM = 2
+    GALAXY = 3
+
+assert IBRAIN.SMALL == 1
+assert int(IBRAIN.SMALL) == 1
+assert IBRAIN.SMALL.value == 1
+assert [thing for thing in IBRAIN] == [1, 2, 3]
+assert list(IBRAIN) == [1, 2, 3]
+assert [thing.value for thing in IBRAIN] == [1, 2, 3]
+assert random.choice(IBRAIN) in [1, 2, 3]  # this still errors but:
+assert random.choice(list(IBRAIN)) in [1, 2, 3]  # this is ok
+
+ +

That’s all fine and good, but I don’t want to use integers. +I want to use strings, because then when I look in my database, +or in printouts, or wherever, +the values will make sense.

+

Well, the docs say +you can just subclass str and make your own StringEnum that will work just like IntEnum. +But it’s LIES:

+
class BRAIN(str, Enum):
+    SMALL = 'small'
+    MEDIUM = 'medium'
+    GALAXY = 'galaxy'
+
+assert BRAIN.SMALL.value == 'small'  # ok, as before
+assert BRAIN.SMALL == 'small'  # yep
+assert list(BRAIN) == ['small', 'medium', 'galaxy']  # hooray!
+assert [thing for thing in BRAIN] == ['small', 'medium', 'galaxy']  # hooray!
+random.choice(BRAIN)  # this still errors but ok i'm getting over it.
+
+# but:
+assert str(BRAIN.SMALL) == 'small'   #NOO!O!O!  'BRAIN.SMALL' != 'small'
+# so, while BRAIN.SMALL == 'small', str(BRAIN.SMALL)  != 'small' aaaargh
+
+ +

So here’s what I ended up with:

+
class BRAIN(str, Enum):
+    SMALL = 'small'
+    MEDIUM = 'medium'
+    GALAXY = 'galaxy'
+
+    def __str__(self) -> str:
+        return str.__str__(self)
+
+ +
    +
  • this basically avoids the need to use .value anywhere at all in your code
  • +
  • enum values duck type to strings in the ways you’d expect
  • +
  • you can iterate over brain and get string-likes out
  • +
  • altho random.choice() is still broken, i leave that as an exercise for the reader
  • +
  • and type hints still work!
  • +
+
# both of these type check ok
+foo = BRAIN.SMALL  # type: str
+bar = BRAIN.SMALL  # type: BRAIN
+
+ +

Example code is in a Gist +if you want to play around. +Let me know if you find anything better!

+
+ +
+ + + + + + + +
+
+ + \ No newline at end of file diff --git a/docs/book/appendix_csvs.html b/book/appendix_csvs.html similarity index 80% rename from docs/book/appendix_csvs.html rename to book/appendix_csvs.html index 7056331..d0b5901 100644 --- a/docs/book/appendix_csvs.html +++ b/book/appendix_csvs.html @@ -3,7 +3,7 @@ - + Swapping Out the Infrastructure: Do Everything with CSVs + @@ -81,7 +183,7 @@
-

Appendix C: Swapping Out the Infrastructure: Do Everything with CSVsDo Everything with CSVs

+

Appendix C: Swapping Out the Infrastructure: Do Everything with CSVs

-

This appendix is intended as a little illustration of the benefits of the +

+This appendix is intended as a little illustration of the benefits of the Repository, Unit of Work, and Service Layer patterns. It’s intended to follow from [chapter_06_uow].

@@ -136,33 +239,31 @@

Appendix C: Swapping Out the Infrastructure: Do Everythin
-
def test_cli_app_reads_csvs_with_batches_and_orders_and_outputs_allocations(
-        make_csv
-):
-    sku1, sku2 = random_ref('s1'), random_ref('s2')
-    batch1, batch2, batch3 = random_ref('b1'), random_ref('b2'), random_ref('b3')
-    order_ref = random_ref('o')
-    make_csv('batches.csv', [
-        ['ref', 'sku', 'qty', 'eta'],
-        [batch1, sku1, 100, ''],
-        [batch2, sku2, 100, '2011-01-01'],
-        [batch3, sku2, 100, '2011-01-02'],
+
def test_cli_app_reads_csvs_with_batches_and_orders_and_outputs_allocations(make_csv):
+    sku1, sku2 = random_ref("s1"), random_ref("s2")
+    batch1, batch2, batch3 = random_ref("b1"), random_ref("b2"), random_ref("b3")
+    order_ref = random_ref("o")
+    make_csv("batches.csv", [
+        ["ref", "sku", "qty", "eta"],
+        [batch1, sku1, 100, ""],
+        [batch2, sku2, 100, "2011-01-01"],
+        [batch3, sku2, 100, "2011-01-02"],
     ])
-    orders_csv = make_csv('orders.csv', [
-        ['orderid', 'sku', 'qty'],
+    orders_csv = make_csv("orders.csv", [
+        ["orderid", "sku", "qty"],
         [order_ref, sku1, 3],
         [order_ref, sku2, 12],
     ])
 
     run_cli_script(orders_csv.parent)
 
-    expected_output_csv = orders_csv.parent / 'allocations.csv'
+    expected_output_csv = orders_csv.parent / "allocations.csv"
     with open(expected_output_csv) as f:
         rows = list(csv.reader(f))
     assert rows == [
-        ['orderid', 'sku', 'qty', 'batchref'],
-        [order_ref, sku1, '3', batch1],
-        [order_ref, sku2, '12', batch2],
+        ["orderid", "sku", "qty", "batchref"],
+        [order_ref, sku1, "3", batch1],
+        [order_ref, sku2, "12", batch2],
     ]
@@ -183,48 +284,46 @@

Appendix C: Swapping Out the Infrastructure: Do Everythin from datetime import datetime from pathlib import Path -from allocation import model +from allocation.domain import model + def load_batches(batches_path): batches = [] with batches_path.open() as inf: reader = csv.DictReader(inf) for row in reader: - if row['eta']: - eta = datetime.strptime(row['eta'], '%Y-%m-%d').date() + if row["eta"]: + eta = datetime.strptime(row["eta"], "%Y-%m-%d").date() else: - eta = None - batches.append(model.Batch( - ref=row['ref'], - sku=row['sku'], - qty=int(row['qty']), - eta=eta - )) + eta = None + batches.append( + model.Batch( + ref=row["ref"], sku=row["sku"], qty=int(row["qty"]), eta=eta + ) + ) return batches - def main(folder): - batches_path = Path(folder) / 'batches.csv' - orders_path = Path(folder) / 'orders.csv' - allocations_path = Path(folder) / 'allocations.csv' + batches_path = Path(folder) / "batches.csv" + orders_path = Path(folder) / "orders.csv" + allocations_path = Path(folder) / "allocations.csv" batches = load_batches(batches_path) - with orders_path.open() as inf, allocations_path.open('w') as outf: + with orders_path.open() as inf, allocations_path.open("w") as outf: reader = csv.DictReader(inf) writer = csv.writer(outf) - writer.writerow(['orderid', 'sku', 'batchref']) + writer.writerow(["orderid", "sku", "batchref"]) for row in reader: - orderid, sku = row['orderid'], row['sku'] - qty = int(row['qty']) + orderid, sku = row["orderid"], row["sku"] + qty = int(row["qty"]) line = model.OrderLine(orderid, sku, qty) batchref = model.allocate(line, batches) writer.writerow([line.orderid, line.sku, batchref]) - -if __name__ == '__main__': +if __name__ == "__main__": main(sys.argv[1])

@@ -244,35 +343,33 @@

Appendix C: Swapping Out the Infrastructure: Do Everythin
-
def test_cli_app_also_reads_existing_allocations_and_can_append_to_them(
-        make_csv
-):
-    sku = random_ref('s')
-    batch1, batch2 = random_ref('b1'), random_ref('b2')
-    old_order, new_order = random_ref('o1'), random_ref('o2')
-    make_csv('batches.csv', [
-        ['ref', 'sku', 'qty', 'eta'],
-        [batch1, sku, 10, '2011-01-01'],
-        [batch2, sku, 10, '2011-01-02'],
+
def test_cli_app_also_reads_existing_allocations_and_can_append_to_them(make_csv):
+    sku = random_ref("s")
+    batch1, batch2 = random_ref("b1"), random_ref("b2")
+    old_order, new_order = random_ref("o1"), random_ref("o2")
+    make_csv("batches.csv", [
+        ["ref", "sku", "qty", "eta"],
+        [batch1, sku, 10, "2011-01-01"],
+        [batch2, sku, 10, "2011-01-02"],
     ])
-    make_csv('allocations.csv', [
-        ['orderid', 'sku', 'qty', 'batchref'],
+    make_csv("allocations.csv", [
+        ["orderid", "sku", "qty", "batchref"],
         [old_order, sku, 10, batch1],
     ])
-    orders_csv = make_csv('orders.csv', [
-        ['orderid', 'sku', 'qty'],
+    orders_csv = make_csv("orders.csv", [
+        ["orderid", "sku", "qty"],
         [new_order, sku, 7],
     ])
 
     run_cli_script(orders_csv.parent)
 
-    expected_output_csv = orders_csv.parent / 'allocations.csv'
+    expected_output_csv = orders_csv.parent / "allocations.csv"
     with open(expected_output_csv) as f:
         rows = list(csv.reader(f))
     assert rows == [
-        ['orderid', 'sku', 'qty', 'batchref'],
-        [old_order, sku, '10', batch1],
-        [new_order, sku, '7', batch2],
+        ["orderid", "sku", "qty", "batchref"],
+        [old_order, sku, "10", batch1],
+        [new_order, sku, "7", batch2],
     ]
@@ -287,9 +384,10 @@

Appendix C: Swapping Out the Infrastructure: Do Everythin with CSVs underlying them instead of a database. And as you’ll see, it really is relatively straightforward.

-

Implementing a Repository and Unit of Work for CSVs

+

Implementing a Repository and Unit of Work for CSVs

-

Here’s what a CSV-based repository could look like. It abstracts away all the +

+Here’s what a CSV-based repository could look like. It abstracts away all the logic for reading CSVs from disk, including the fact that it has to read two different CSVs (one for batches and one for allocations), and it gives us just the familiar .list() API, which provides the illusion of an in-memory @@ -301,10 +399,9 @@

Implementing a Rep
class CsvRepository(repository.AbstractRepository):
-
     def __init__(self, folder):
-        self._batches_path = Path(folder) / 'batches.csv'
-        self._allocations_path = Path(folder) / 'allocations.csv'
+        self._batches_path = Path(folder) / "batches.csv"
+        self._allocations_path = Path(folder) / "allocations.csv"
         self._batches = {}  # type: Dict[str, model.Batch]
         self._load()
 
@@ -318,22 +415,20 @@ 

Implementing a Rep with self._batches_path.open() as f: reader = csv.DictReader(f) for row in reader: - ref, sku = row['ref'], row['sku'] - qty = int(row['qty']) - if row['eta']: - eta = datetime.strptime(row['eta'], '%Y-%m-%d').date() + ref, sku = row["ref"], row["sku"] + qty = int(row["qty"]) + if row["eta"]: + eta = datetime.strptime(row["eta"], "%Y-%m-%d").date() else: - eta = None - self._batches[ref] = model.Batch( - ref=ref, sku=sku, qty=qty, eta=eta - ) - if self._allocations_path.exists() is False: + eta = None + self._batches[ref] = model.Batch(ref=ref, sku=sku, qty=qty, eta=eta) + if self._allocations_path.exists() is False: return with self._allocations_path.open() as f: reader = csv.DictReader(f) for row in reader: - batchref, orderid, sku = row['batchref'], row['orderid'], row['sku'] - qty = int(row['qty']) + batchref, orderid, sku = row["batchref"], row["orderid"], row["sku"] + qty = int(row["qty"]) line = model.OrderLine(orderid, sku, qty) batch = self._batches[batchref] batch._allocations.add(line) @@ -345,7 +440,8 @@

Implementing a Rep

-

And here’s what a UoW for CSVs would look like:

+

+And here’s what a UoW for CSVs would look like:

A UoW for CSVs: commit = csv.writer (src/allocation/service_layer/csv_uow.py)
@@ -353,14 +449,13 @@

Implementing a Rep
class CsvUnitOfWork(unit_of_work.AbstractUnitOfWork):
-
     def __init__(self, folder):
         self.batches = CsvRepository(folder)
 
     def commit(self):
-        with self.batches._allocations_path.open('w') as f:
+        with self.batches._allocations_path.open("w") as f:
             writer = csv.writer(f)
-            writer.writerow(['orderid', 'sku', 'qty', 'batchref'])
+            writer.writerow(["orderid", "sku", "qty", "batchref"])
             for batch in self.batches.list():
                 for line in batch._allocations:
                     writer.writerow(
@@ -385,20 +480,21 @@ 

Implementing a Rep
def main(folder):
-    orders_path = Path(folder) / 'orders.csv'
+    orders_path = Path(folder) / "orders.csv"
     uow = csv_uow.CsvUnitOfWork(folder)
     with orders_path.open() as f:
         reader = csv.DictReader(f)
         for row in reader:
-            orderid, sku = row['orderid'], row['sku']
-            qty = int(row['qty'])
+            orderid, sku = row["orderid"], row["sku"]
+            qty = int(row["qty"])
             services.allocate(orderid, sku, qty, uow)

-

Ta-da! Now are y’all impressed or what?

+

+Ta-da! Now are y’all impressed or what?

Much love,

@@ -409,96 +505,30 @@

Implementing a Rep

+
-
diff --git a/docs/book/appendix_django.html b/book/appendix_django.html similarity index 80% rename from docs/book/appendix_django.html rename to book/appendix_django.html index b96fbce..6c9a224 100644 --- a/docs/book/appendix_django.html +++ b/book/appendix_django.html @@ -3,7 +3,7 @@ - + Repository and Unit of Work Patterns with Django + @@ -81,7 +183,7 @@

-

Appendix D: Repository and Unit of Work Patterns with DjangoPatterns with Django

+

Appendix D: Repository and Unit of Work Patterns with Django

-

Suppose you wanted to use Django instead of SQLAlchemy and Flask. How +

+ +Suppose you wanted to use Django instead of SQLAlchemy and Flask. How might things look? The first thing is to choose where to install it. We put it in a separate package next to our main allocation code:

@@ -118,35 +222,35 @@

Appendix D: Repository and Unit of Work Patterns with D
-
├── src
-│   ├── allocation
-│   │   ├── __init__.py
-│   │   ├── adapters
-│   │   │   ├── __init__.py
-...
-│   ├── djangoproject
-│   │   ├── alloc
-│   │   │   ├── __init__.py
-│   │   │   ├── apps.py
-│   │   │   ├── migrations
-│   │   │   │   ├── 0001_initial.py
-│   │   │   │   └── __init__.py
-│   │   │   ├── models.py
-│   │   │   └── views.py
-│   │   ├── django_project
-│   │   │   ├── __init__.py
-│   │   │   ├── settings.py
-│   │   │   ├── urls.py
-│   │   │   └── wsgi.py
-│   │   └── manage.py
-│   └── setup.py
-└── tests
-    ├── conftest.py
-    ├── e2e
-    │   └── test_api.py
-    ├── integration
-    │   ├── test_repository.py
-...
+
├── src
+│   ├── allocation
+│   │   ├── __init__.py
+│   │   ├── adapters
+│   │   │   ├── __init__.py
+...
+│   ├── djangoproject
+│   │   ├── alloc
+│   │   │   ├── __init__.py
+│   │   │   ├── apps.py
+│   │   │   ├── migrations
+│   │   │   │   ├── 0001_initial.py
+│   │   │   │   └── __init__.py
+│   │   │   ├── models.py
+│   │   │   └── views.py
+│   │   ├── django_project
+│   │   │   ├── __init__.py
+│   │   │   ├── settings.py
+│   │   │   ├── urls.py
+│   │   │   └── wsgi.py
+│   │   └── manage.py
+│   └── setup.py
+└── tests
+    ├── conftest.py
+    ├── e2e
+    │   └── test_api.py
+    ├── integration
+    │   ├── test_repository.py
+...
@@ -169,14 +273,20 @@

Appendix D: Repository and Unit of Work Patterns with D git checkout appendix_django

+
+

Code examples follows on from the end of [chapter_06_uow].

+
-

Repository Pattern with Django

+

Repository Pattern with Django

-

We used a plug-in called +

+ + +We used a plugin called pytest-django to help with test database management.

@@ -192,7 +302,7 @@

Repository Pattern with Django

from djangoproject.alloc import models as django_models
 
 
-@pytest.mark.django_db
+@pytest.mark.django_db
 def test_repository_can_save_a_batch():
     batch = model.Batch("batch1", "RUSTY-SOAPDISH", 100, eta=date(2011, 12, 25))
 
@@ -217,22 +327,22 @@ 

Repository Pattern with Django

-
@pytest.mark.django_db
+
@pytest.mark.django_db
 def test_repository_can_retrieve_a_batch_with_allocations():
     sku = "PONY-STATUE"
     d_line = django_models.OrderLine.objects.create(orderid="order1", sku=sku, qty=12)
     d_batch1 = django_models.Batch.objects.create(
-        reference="batch1", sku=sku, qty=100, eta=None
+        reference="batch1", sku=sku, qty=100, eta=None
     )
     d_batch2 = django_models.Batch.objects.create(
-        reference="batch2", sku=sku, qty=100, eta=None
+        reference="batch2", sku=sku, qty=100, eta=None
     )
     django_models.Allocation.objects.create(line=d_line, batch=d_batch1)
 
     repo = repository.DjangoRepository()
     retrieved = repo.get("batch1")
 
-    expected = model.Batch("batch1", sku, 100, eta=None)
+    expected = model.Batch("batch1", sku, 100, eta=None)
     assert retrieved == expected  # Batch.__eq__ only compares reference
     assert retrieved.sku == expected.sku
     assert retrieved._purchased_quantity == expected._purchased_quantity
@@ -252,7 +362,6 @@ 

Repository Pattern with Django

class DjangoRepository(AbstractRepository):
-
     def add(self, batch):
         super().add(batch)
         self.update(batch)
@@ -261,9 +370,11 @@ 

Repository Pattern with Django

django_models.Batch.update_from_domain(batch) def _get(self, reference): - return django_models.Batch.objects.filter( - reference=reference - ).first().to_domain() + return ( + django_models.Batch.objects.filter(reference=reference) + .first() + .to_domain() + ) def list(self): return [b.to_domain() for b in django_models.Batch.objects.all()]
@@ -276,9 +387,11 @@

Repository Pattern with Django

some custom methods for translating to and from our domain model.[1]

-

Custom Methods on Django ORM Classes to Translate to/from Our Domain Model

+

Custom Methods on Django ORM Classes to Translate to/from Our Domain Model

-

Those custom methods look something like this:

+

+ +Those custom methods look something like this:

Django ORM with custom methods for domain model conversion (src/djangoproject/alloc/models.py)
@@ -288,11 +401,12 @@

from django.db import models from allocation.domain import model as domain_model + class Batch(models.Model): reference = models.CharField(max_length=255) sku = models.CharField(max_length=255) qty = models.IntegerField() - eta = models.DateField(blank=True, null=True) + eta = models.DateField(blank=True, null=True) @staticmethod def update_from_domain(batch: domain_model.Batch): @@ -351,6 +465,8 @@

As in [chapter_02_repository], we use dependency inversion. The ORM (Django) depends on the model and not the other way around. + + @@ -358,9 +474,11 @@

-

Unit of Work Pattern with Django

+

Unit of Work Pattern with Django

-

The tests don’t change too much:

+

+ +The tests don’t change too much:

Adapted UoW tests (tests/integration/test_uow.py)
@@ -370,32 +488,33 @@

Unit of Work Pattern with Django

def insert_batch(ref, sku, qty, eta):  #(1)
     django_models.Batch.objects.create(reference=ref, sku=sku, qty=qty, eta=eta)
 
+
 def get_allocated_batch_ref(orderid, sku):  #(1)
     return django_models.Allocation.objects.get(
         line__orderid=orderid, line__sku=sku
     ).batch.reference
 
 
-@pytest.mark.django_db(transaction=True)
+@pytest.mark.django_db(transaction=True)
 def test_uow_can_retrieve_a_batch_and_allocate_to_it():
-    insert_batch('batch1', 'HIPSTER-WORKBENCH', 100, None)
+    insert_batch("batch1", "HIPSTER-WORKBENCH", 100, None)
 
     uow = unit_of_work.DjangoUnitOfWork()
     with uow:
-        batch = uow.batches.get(reference='batch1')
-        line = model.OrderLine('o1', 'HIPSTER-WORKBENCH', 10)
+        batch = uow.batches.get(reference="batch1")
+        line = model.OrderLine("o1", "HIPSTER-WORKBENCH", 10)
         batch.allocate(line)
         uow.commit()
 
-    batchref = get_allocated_batch_ref('o1', 'HIPSTER-WORKBENCH')
-    assert batchref == 'batch1'
+    batchref = get_allocated_batch_ref("o1", "HIPSTER-WORKBENCH")
+    assert batchref == "batch1"
 
 
-@pytest.mark.django_db(transaction=True)  #(2)
+@pytest.mark.django_db(transaction=True)  #(2)
 def test_rolls_back_uncommitted_work_by_default():
     ...
 
-@pytest.mark.django_db(transaction=True)  #(2)
+@pytest.mark.django_db(transaction=True)  #(2)
 def test_rolls_back_on_error():
     ...
@@ -426,15 +545,14 @@

Unit of Work Pattern with Django

class DjangoUnitOfWork(AbstractUnitOfWork):
-
     def __enter__(self):
         self.batches = repository.DjangoRepository()
-        transaction.set_autocommit(False)  #(1)
+        transaction.set_autocommit(False)  #(1)
         return super().__enter__()
 
     def __exit__(self, *args):
         super().__exit__(*args)
-        transaction.set_autocommit(True)
+        transaction.set_autocommit(True)
 
     def commit(self):
         for batch in self.batches.seen:  #(3)
@@ -462,15 +580,21 @@ 

Unit of Work Pattern with Django

instrumenting the domain model instances themselves, the commit() command needs to explicitly go through all the objects that have been touched by every repository and manually -update them back to the ORM.

+update them back to the ORM. + +

-

API: Django Views Are Adapters

+

API: Django Views Are Adapters

-

The Django views.py file ends up being almost identical to the +

+ + + +The Django views.py file ends up being almost identical to the old flask_app.py, because our architecture means it’s a very thin wrapper around our service layer (which didn’t change at all, by the way):

@@ -479,44 +603,47 @@

API: Django Views Are Adapters

-
os.environ['DJANGO_SETTINGS_MODULE'] = 'djangoproject.django_project.settings'
+
os.environ["DJANGO_SETTINGS_MODULE"] = "djangoproject.django_project.settings"
 django.setup()
 
+
 @csrf_exempt
 def add_batch(request):
     data = json.loads(request.body)
-    eta = data['eta']
-    if eta is not None:
+    eta = data["eta"]
+    if eta is not None:
         eta = datetime.fromisoformat(eta).date()
     services.add_batch(
-        data['ref'], data['sku'], data['qty'], eta,
+        data["ref"], data["sku"], data["qty"], eta,
         unit_of_work.DjangoUnitOfWork(),
     )
-    return HttpResponse('OK', status=201)
+    return HttpResponse("OK", status=201)
+
 
 @csrf_exempt
 def allocate(request):
     data = json.loads(request.body)
     try:
         batchref = services.allocate(
-            data['orderid'],
-            data['sku'],
-            data['qty'],
+            data["orderid"],
+            data["sku"],
+            data["qty"],
             unit_of_work.DjangoUnitOfWork(),
         )
     except (model.OutOfStock, services.InvalidSku) as e:
-        return JsonResponse({'message': str(e)}, status=400)
+        return JsonResponse({"message": str(e)}, status=400)
 
-    return JsonResponse({'batchref': batchref}, status=201)
+ return JsonResponse({"batchref": batchref}, status=201)

-

Why Was This All So Hard?

+

Why Was This All So Hard?

-

OK, it works, but it does feel like more effort than Flask/SQLAlchemy. Why is +

+OK, it works, but it does feel like more effort than Flask/SQLAlchemy. Why is that?

@@ -528,7 +655,8 @@

Why Was This All So Hard?

high).

-

Because Django is so tightly coupled to the database, you have to use helpers +

+Because Django is so tightly coupled to the database, you have to use helpers like pytest-django and think carefully about test databases, right from the very first line of code, in a way that we didn’t have to when we started out with our pure domain model.

@@ -547,9 +675,10 @@

Why Was This All So Hard?

-

What to Do If You Already Have Django

+

What to Do If You Already Have Django

-

So what should you do if you want to apply some of the patterns in this book +

+So what should you do if you want to apply some of the patterns in this book to a Django app? We’d say the following:

@@ -587,9 +716,10 @@

What to Do If You Already Have D

-

Steps Along the Way

+

Steps Along the Way

-

Suppose you’re working on a Django project that you’re not sure is going +

+Suppose you’re working on a Django project that you’re not sure is going to get complex enough to warrant the patterns we recommend, but you still want to put a few steps in place to make your life easier, both in the medium term and if you want to migrate to some of our patterns later. Consider the following:

@@ -636,12 +766,18 @@

Steps Along the Way

-

For more thoughts and actual lived experience dealing with existing +

+For more thoughts and actual lived experience dealing with existing applications, refer to the epilogue.

+

@@ -654,93 +790,22 @@

Steps Along the Way

-
diff --git a/docs/book/appendix_ds1_table.html b/book/appendix_ds1_table.html similarity index 83% rename from docs/book/appendix_ds1_table.html rename to book/appendix_ds1_table.html index 96d9e5f..3ef04e4 100644 --- a/docs/book/appendix_ds1_table.html +++ b/book/appendix_ds1_table.html @@ -3,7 +3,7 @@ - + Summary Diagram and Table + @@ -81,7 +183,7 @@
-

Appendix B: A Template Project Structure

+

Appendix B: A Template Project Structure

-

Around [chapter_04_service_layer], we moved from just having +

+Around [chapter_04_service_layer], we moved from just having everything in one folder to a more structured tree, and we thought it might be of interest to outline the moving parts.

@@ -235,7 +338,7 @@

Appendix B: A Template Project Structure

Let’s look at a few of these files and concepts in more detail.

-

Env Vars, 12-Factor, and Config, Inside and Outside Containers

+

Env Vars, 12-Factor, and Config, Inside and Outside Containers

The basic problem we’re trying to solve here is that we need different config settings for the following:

@@ -261,7 +364,7 @@

Env Vars,

-

Config.py

+

Config.py

Whenever our application code needs access to some config, it’s going to get it from a file called config.py. Here are a couple of examples from our @@ -274,18 +377,19 @@

Config.py

import os
 
+
 def get_postgres_uri():  #(1)
-    host = os.environ.get('DB_HOST', 'localhost')  #(2)
-    port = 54321 if host == 'localhost' else 5432
-    password = os.environ.get('DB_PASSWORD', 'abc123')
-    user, db_name = 'allocation', 'allocation'
-    return f"postgresql://{user}:{password}@{host}:{port}/{db_name}"
+    host = os.environ.get("DB_HOST", "localhost")  #(2)
+    port = 54321 if host == "localhost" else 5432
+    password = os.environ.get("DB_PASSWORD", "abc123")
+    user, db_name = "allocation", "allocation"
+    return f"postgresql://{user}:{password}@{host}:{port}/{db_name}"
 
 
 def get_api_url():
-    host = os.environ.get('API_HOST', 'localhost')
-    port = 5005 if host == 'localhost' else 80
-    return f"http://{host}:{port}"
+ host = os.environ.get("API_HOST", "localhost") + port = 5005 if host == "localhost" else 80 + return f"http://{host}:{port}"
@@ -325,7 +429,7 @@

Config.py

-

Docker-Compose and Containers Config

+

Docker-Compose and Containers Config

We use a lightweight Docker container orchestration tool called docker-compose. It’s main configuration is via a YAML file (sigh):[5]

@@ -335,34 +439,34 @@

Docker-Compose and Containers Con
-
version: "3"
-services:
-
-  app:  #(1)
-    build:
-      context: .
-      dockerfile: Dockerfile
-    depends_on:
-      - postgres
-    environment:  #(3)
-      - DB_HOST=postgres  (4)
-      - DB_PASSWORD=abc123
-      - API_HOST=app
-      - PYTHONDONTWRITEBYTECODE=1  #(5)
-    volumes:  #(6)
-      - ./src:/src
-      - ./tests:/tests
-    ports:
-      - "5005:80"  (7)
-
-
-  postgres:
-    image: postgres:9.6  #(2)
-    environment:
-      - POSTGRES_USER=allocation
-      - POSTGRES_PASSWORD=abc123
-    ports:
-      - "54321:5432"
+
version: "3"
+services:
+
+  app:  #(1)
+    build:
+      context: .
+      dockerfile: Dockerfile
+    depends_on:
+      - postgres
+    environment:  #(3)
+      - DB_HOST=postgres  (4)
+      - DB_PASSWORD=abc123
+      - API_HOST=app
+      - PYTHONDONTWRITEBYTECODE=1  #(5)
+    volumes:  #(6)
+      - ./src:/src
+      - ./tests:/tests
+    ports:
+      - "5005:80"  (7)
+
+
+  postgres:
+    image: postgres:9.6  #(2)
+    environment:
+      - POSTGRES_USER=allocation
+      - POSTGRES_PASSWORD=abc123
+    ports:
+      - "54321:5432"
@@ -426,7 +530,7 @@

Docker-Compose and Containers Con

-

Installing Your Source as a Package

+

Installing Your Source as a Package

All our application code (everything except tests, really) lives inside an src folder:

@@ -463,9 +567,7 @@

Installing Your Source as a Packag
from setuptools import setup
 
 setup(
-    name='allocation',
-    version='0.1',
-    packages=['allocation'],
+    name="allocation", version="0.1", packages=["allocation"],
 )

@@ -479,7 +581,7 @@

Installing Your Source as a Packag
-

Dockerfile

+

Dockerfile

Dockerfiles are going to be very project-specific, but here are a few key stages you’ll expect to see:

@@ -489,28 +591,25 @@

Dockerfile

-
FROM python:3.8-alpine
+
FROM python:3.9-slim-buster
 
 (1)
-RUN apk add --no-cache --virtual .build-deps gcc postgresql-dev musl-dev python3-dev
-RUN apk add libpq
+# RUN apt install gcc libpq (no longer needed bc we use psycopg2-binary)
 
 (2)
-COPY requirements.txt /tmp/
-RUN pip install -r /tmp/requirements.txt
-
-RUN apk del --no-cache .build-deps
+COPY requirements.txt /tmp/
+RUN pip install -r /tmp/requirements.txt
 
 (3)
-RUN mkdir -p /src
-COPY src/ /src/
-RUN pip install -e /src
-COPY tests/ /tests/
+RUN mkdir -p /src
+COPY src/ /src/
+RUN pip install -e /src
+COPY tests/ /tests/
 
 (4)
-WORKDIR /src
-ENV FLASK_APP=allocation/entrypoints/flask_app.py FLASK_DEBUG=1 PYTHONUNBUFFERED=1
-CMD flask run --host=0.0.0.0 --port=80
+WORKDIR /src +ENV FLASK_APP=allocation/entrypoints/flask_app.py FLASK_DEBUG=1 PYTHONUNBUFFERED=1 +CMD flask run --host=0.0.0.0 --port=80
@@ -551,9 +650,10 @@

Dockerfile

-

Tests

+

Tests

-

Our tests are kept alongside everything else, as shown here:

+

+Our tests are kept alongside everything else, as shown here:

Tests folder tree
@@ -588,7 +688,7 @@

Tests

-

Wrap-Up

+

Wrap-Up

These are our basic building blocks:

@@ -609,12 +709,18 @@

Wrap-Up

-

We doubt that anyone will end up with exactly the same solutions we did, but we hope you +

+We doubt that anyone will end up with exactly the same solutions we did, but we hope you find some inspiration here.

+

@@ -642,93 +748,22 @@

Wrap-Up

-
diff --git a/docs/book/appendix_validation.html b/book/appendix_validation.html similarity index 84% rename from docs/book/appendix_validation.html rename to book/appendix_validation.html index 00d7322..9713ce3 100644 --- a/docs/book/appendix_validation.html +++ b/book/appendix_validation.html @@ -3,7 +3,7 @@ - + Validation + @@ -81,7 +183,7 @@
-

Appendix E: Validation

+

Appendix E: Validation

-

Whenever we’re teaching and talking about these techniques, one question that +

+Whenever we’re teaching and talking about these techniques, one question that comes up over and over is "Where should I do validation? Does that belong with my business logic in the domain model, or is that an infrastructural concern?"

@@ -123,7 +226,7 @@

Appendix E: Validation

with irrelevant detail.

-

What Is Validation, Anyway?

+

What Is Validation, Anyway?

When people use the word validation, they usually mean a process whereby they test the inputs of an operation to make sure that they match certain criteria. @@ -138,7 +241,7 @@

What Is Validation, Anyway?

-

Validating Syntax

+

Validating Syntax

In linguistics, the syntax of a language is the set of rules that govern the structure of grammatical sentences. For example, in English, the sentence @@ -187,10 +290,10 @@

Validating Syntax

class Allocate(Command): _schema = Schema({ #(1) - 'orderid': int, - sku: str, - qty: And(Use(int), lambda n: n > 0) - }, ignore_extra_keys=True) + 'orderid': str, + 'sku': str, + 'qty': And(Use(int), lambda n: n > 0) + }, ignore_extra_keys=True) orderid: str sku: str @@ -198,8 +301,8 @@

Validating Syntax

@classmethod def from_json(cls, data): #(2) - data = json.loads(data) - return cls(**_schema.validate(data)) + data = json.loads(data) + return cls(**_schema.validate(data))
@@ -227,8 +330,8 @@

Validating Syntax

def command(name, **fields):  #(1)
-    schema = Schema(And(Use(json.loads), fields), ignore_extra_keys=True)  #(2)
-    cls = make_dataclass(name, fields.keys())
+    schema = Schema(And(Use(json.loads), fields), ignore_extra_keys=True)
+    cls = make_dataclass(name, fields.keys())  #(2)
     cls.from_json = lambda s: cls(**schema.validate(s))  #(3)
     return cls
 
@@ -238,12 +341,14 @@ 

Validating Syntax

quantity = And(Use(int), greater_than_zero) #(4) Allocate = command( #(5) + 'Allocate', orderid=int, sku=str, qty=quantity ) AddStock = command( + 'AddStock', sku=str, qty=quantity
@@ -278,7 +383,7 @@

Validating Syntax

-

Postel’s Law and the Tolerant Reader Pattern

+

Postel’s Law and the Tolerant Reader Pattern

Postel’s law, or the robustness principle, tells us, "Be liberal in what you accept, and conservative in what you emit." We think this applies particularly @@ -367,7 +472,7 @@

Postel’s Law and the

-

Validating at the Edge

+

Validating at the Edge

Earlier, we said that we want to avoid cluttering our code with irrelevant details. In particular, we don’t want to code defensively inside our domain model. @@ -400,12 +505,12 @@

Validating at the Edge

message = message_type.from_json(body) self.handle([message]) except StopIteration: - raise KeyError(f"Unknown message name {name}") + raise KeyError(f"Unknown message name {name}") except ValidationError as e: logging.error( - f'invalid message of type {name}\n' - f'{body}\n' - f'{e}' + f'invalid message of type {name}\n' + f'{body}\n' + f'{e}' ) raise e
@@ -420,7 +525,7 @@

Validating at the Edge

-
@app.route("/change_quantity", methods=['POST'])
+
@app.route("/change_quantity", methods=['POST'])
 def change_batch_quantity():
     try:
         bus.handle_message('ChangeBatchQuantity', request.body)
@@ -447,9 +552,9 @@ 

Validating at the Edge

try: bus.handle_message('ChangeBatchQuantity', m) except ValidationError: - print('Skipping invalid message') + print('Skipping invalid message') except exceptions.InvalidSku as e: - print(f'Unable to change stock for missing sku {e}')
+ print(f'Unable to change stock for missing sku {e}')
@@ -478,7 +583,7 @@

Validating at the Edge

-

Validating Semantics

+

Validating Semantics

While syntax is concerned with the structure of messages, semantics is the study of meaning in messages. The sentence "Undo no dogs from ellipsis four" is @@ -522,10 +627,10 @@

Validating Semantics

self.message = message class ProductNotFound(MessageUnprocessable): #(2) - """" - This exception is raised when we try to perform an action on a product - that doesn't exist in our database. - """" + """" + This exception is raised when we try to perform an action on a product + that doesn't exist in our database. + """" def __init__(self, message): super().__init__(message) @@ -533,7 +638,7 @@

Validating Semantics

def product_exists(event, uow): #(3) product = uow.products.get(event.sku) - if product is None: + if product is None: raise ProductNotFound(event)
@@ -568,9 +673,9 @@

Validating Semantics

from allocation import ensure def allocate(event, uow): - line = mode.OrderLine(event.orderid, event.sku, event.qty) + line = model.OrderLine(event.orderid, event.sku, event.qty) with uow: - ensure.product_exists(uow, event) + ensure.product_exists(event, uow) product = uow.products.get(line.sku) product.allocate(line) @@ -594,7 +699,7 @@

Validating Semantics

class SkipMessage (Exception):
-    """"
+    """"
     This exception is raised when a message can't be processed, but there's no
     incorrect behavior. For example, we might receive the same message multiple
     times, or we might receive a message that is now out of date.
@@ -605,8 +710,8 @@ 

Validating Semantics

def batch_is_new(self, event, uow): batch = uow.batches.get(event.batchid) - if batch is not None: - raise SkipMessage(f"Batch with id {event.batchid} already exists")
+ if batch is not None: + raise SkipMessage(f"Batch with id {event.batchid} already exists")
@@ -624,9 +729,9 @@

Validating Semantics

def handle_message(self, message): try: - ... - except SkipMessage as e: - logging.warn(f"Skipping message {message.id} because {e.reason}") + ... + except SkipMessage as e: + logging.warn(f"Skipping message {message.id} because {e.reason}") @@ -643,7 +748,7 @@

Validating Semantics

-

Validating Pragmatics

+

Validating Pragmatics

Pragmatics is the study of how we understand language in context. After we have parsed a message and grasped its meaning, we still need to process it in @@ -713,101 +818,35 @@

Validating Pragmatics

domain model. When we receive a message like "allocate three million units of SCARCE-CLOCK to order 76543," the message is syntactically valid and semantically valid, but we’re unable to comply because we don’t have the stock -available.

+available. +

+ -
diff --git a/docs/book/author_bio.html b/book/author_bio.html similarity index 100% rename from docs/book/author_bio.html rename to book/author_bio.html diff --git a/docs/book/book.html b/book/book.html similarity index 100% rename from docs/book/book.html rename to book/book.html diff --git a/docs/book/chapter_01_domain_model.html b/book/chapter_01_domain_model.html similarity index 83% rename from docs/book/chapter_01_domain_model.html rename to book/chapter_01_domain_model.html index 8adecf0..5f0da22 100644 --- a/docs/book/chapter_01_domain_model.html +++ b/book/chapter_01_domain_model.html @@ -3,7 +3,7 @@ - + Domain Modeling + @@ -81,7 +183,7 @@
-

Domain Modeling

+

1: Domain Modeling

-

This chapter looks into how we can model business processes with code, in a way +

+ +This chapter looks into how we can model business processes with code, in a way that’s highly compatible with TDD. We’ll discuss why domain modeling matters, and we’ll look at a few key patterns for modeling domains: Entity, Value Object, and Domain Service.

@@ -128,31 +232,34 @@

Domain Modeling

Figure 1. A placeholder illustration of our domain model
-

What Is a Domain Model?

+

What Is a Domain Model?

-

In the introduction, we used the term business logic layer to describe the -central layer of a three-layered architecture. For the rest of the book, we’re -going to use the term domain model instead. This is a term from the DDD -community that does a better job of capturing our intended meaning (see the -next sidebar for more on DDD).

+

+In the introduction, we used the term business logic layer +to describe the central layer of a three-layered architecture. For the rest of +the book, we’re going to use the term domain model instead. This is a term +from the DDD community that does a better job of capturing our intended meaning +(see the next sidebar for more on DDD).

-

The domain is a fancy way of saying the problem you’re trying to solve. Your -authors currently work for an online retailer of furniture. Depending on which system -you’re talking about, the domain might be purchasing and procurement, or product -design, or logistics and delivery. Most programmers spend their days trying to -improve or automate business processes; the domain is the set of activities -that those processes support.

+

+The domain is a fancy way of saying the problem you’re trying to solve. +Your authors currently work for an online retailer of furniture. Depending on +which system you’re talking about, the domain might be purchasing and +procurement, or product design, or logistics and delivery. Most programmers +spend their days trying to improve or automate business processes; the domain +is the set of activities that those processes support.

-

A model is a map of a process or phenomenon that captures a useful property. +

+A model is a map of a process or phenomenon that captures a useful property. Humans are exceptionally good at producing models of things in their heads. For example, when someone throws a ball toward you, you’re able to predict its -movement almost unconsciously, because you have a model of the way objects move in -space. Your model isn’t perfect by any means. Humans have terrible intuitions -about how objects behave at near-light speeds or in a vacuum because our model -was never designed to cover those cases. That doesn’t mean the model is wrong, -but it does mean that some predictions fall outside of its domain.

+movement almost unconsciously, because you have a model of the way objects move +in space. Your model isn’t perfect by any means. Humans have terrible +intuitions about how objects behave at near-light speeds or in a vacuum because +our model was never designed to cover those cases. That doesn’t mean the model +is wrong, but it does mean that some predictions fall outside of its domain.

The domain model is the mental map that business owners have of their @@ -189,7 +296,8 @@

What Is a Domain Model?

Domain-driven design, or DDD, popularized the concept of domain modeling,[1] and it’s been a hugely successful movement in transforming the way people design software by focusing on the core business domain. Many of the -architecture patterns that we cover in this book—including Entity, Aggregate, Value Object (see [chapter_07_aggregate]), and Repository (in +architecture patterns that we cover in this book—including Entity, Aggregate, +Value Object (see [chapter_07_aggregate]), and Repository (in the next chapter)—come from the DDD tradition.

@@ -301,9 +409,11 @@

What Is a Domain Model?

-

Exploring the Domain Language

+

Exploring the Domain Language

-

Understanding the domain model takes time, and patience, and Post-it notes. We +

+ +Understanding the domain model takes time, and patience, and Post-it notes. We have an initial conversation with our business experts and agree on a glossary and some rules for the first minimal version of the domain model. Wherever possible, we ask for concrete examples to illustrate each rule.

@@ -314,7 +424,7 @@

Exploring the Domain Language

so that the examples are easier to talk about.

-

#allocation_notes shows some notes we might have taken while having a +

The following sidebar shows some notes we might have taken while having a conversation with our domain experts about allocation.

@@ -386,9 +496,11 @@

Exploring the Domain Language

-

Unit Testing Domain Models

+

Unit Testing Domain Models

-

We’re not going to show you how TDD works in this book, but we want to show you +

+ +We’re not going to show you how TDD works in this book, but we want to show you how we would construct a model from this business conversation.

@@ -397,7 +509,7 @@

Unit Testing Domain Models

Why not have a go at solving this problem yourself? Write a few unit tests to see if you can capture the essence of these business rules in nice, clean -code.

+code (ideally without looking at the solution we came up with below!)

You’ll find some placeholder unit tests on GitHub, but you could just start from @@ -415,7 +527,7 @@

Unit Testing Domain Models

def test_allocating_to_a_batch_reduces_the_available_quantity():
     batch = Batch("batch-001", "SMALL-TABLE", qty=20, eta=date.today())
-    line = OrderLine('order-ref', "SMALL-TABLE", 2)
+    line = OrderLine("order-ref", "SMALL-TABLE", 2)
 
     batch.allocate(line)
 
@@ -438,7 +550,7 @@ 

Unit Testing Domain Models

-
@dataclass(frozen=True)  #(1) (2)
+
@dataclass(frozen=True)  #(1) (2)
 class OrderLine:
     orderid: str
     sku: str
@@ -446,16 +558,14 @@ 

Unit Testing Domain Models

class Batch: - def __init__( - self, ref: str, sku: str, qty: int, eta: Optional[date] #(2) - ): + def __init__(self, ref: str, sku: str, qty: int, eta: Optional[date]): #(2) self.reference = ref self.sku = sku self.eta = eta self.available_quantity = qty - def allocate(self, line: OrderLine): - self.available_quantity -= line.qty #(3)
+ def allocate(self, line: OrderLine): #(3) + self.available_quantity -= line.qty
@@ -473,21 +583,23 @@

Unit Testing Domain Models

typing.Optional and datetime.date. If you want to double-check anything, you can see the full working code for each chapter in its branch (e.g., -chapter_01_domain_model).

+chapter_01_domain_model).

  • Type hints are still a matter of controversy in the Python world. For domain models, they can sometimes help to clarify or document what the expected arguments are, and people with IDEs are often grateful for them. -You may decide the price paid in terms of readability is too high.

    +You may decide the price paid in terms of readability is too high. +

  • -

    Our implementation here is trivial: a Batch just wraps an integer -available_quantity, and we decrement that value on allocation. We’ve written -quite a lot of code just to subtract one number from another, but we think that modeling our -domain precisely will pay off.[3].]

    +

    Our implementation here is trivial: +a Batch just wraps an integer available_quantity, +and we decrement that value on allocation. +We’ve written quite a lot of code just to subtract one number from another, +but we think that modeling our domain precisely will pay off.[3]

    Let’s write some new failing tests:

    @@ -500,26 +612,25 @@

    Unit Testing Domain Models

    def make_batch_and_line(sku, batch_qty, line_qty):
         return (
             Batch("batch-001", sku, batch_qty, eta=date.today()),
    -        OrderLine("order-123", sku, line_qty)
    +        OrderLine("order-123", sku, line_qty),
         )
     
    -
     def test_can_allocate_if_available_greater_than_required():
         large_batch, small_line = make_batch_and_line("ELEGANT-LAMP", 20, 2)
         assert large_batch.can_allocate(small_line)
     
     def test_cannot_allocate_if_available_smaller_than_required():
         small_batch, large_line = make_batch_and_line("ELEGANT-LAMP", 2, 20)
    -    assert small_batch.can_allocate(large_line) is False
    +    assert small_batch.can_allocate(large_line) is False
     
     def test_can_allocate_if_available_equal_to_required():
         batch, line = make_batch_and_line("ELEGANT-LAMP", 2, 2)
         assert batch.can_allocate(line)
     
     def test_cannot_allocate_if_skus_do_not_match():
    -    batch = Batch("batch-001", "UNCOMFORTABLE-CHAIR", 100, eta=None)
    +    batch = Batch("batch-001", "UNCOMFORTABLE-CHAIR", 100, eta=None)
         different_sku_line = OrderLine("order-123", "EXPENSIVE-TOASTER", 10)
    -    assert batch.can_allocate(different_sku_line) is False
    + assert batch.can_allocate(different_sku_line) is False
    @@ -576,9 +687,7 @@

    Unit Testing Domain Models

    class Batch:
    -    def __init__(
    -        self, ref: str, sku: str, qty: int, eta: Optional[date]
    -    ):
    +    def __init__(self, ref: str, sku: str, qty: int, eta: Optional[date]):
             self.reference = ref
             self.sku = sku
             self.eta = eta
    @@ -685,8 +794,9 @@ 

    Unit Testing Domain Models

    than we can show on the page!

    -

    But taking this simple domain model as a placeholder for something more complex, we’re going to extend our simple domain model in the rest of the book and -plug it into the real world of APIs and databases and spreadsheets. We’ll +

    But taking this simple domain model as a placeholder for something more +complex, we’re going to extend our simple domain model in the rest of the book +and plug it into the real world of APIs and databases and spreadsheets. We’ll see how sticking rigidly to our principles of encapsulation and careful layering will help us to avoid a ball of mud.

    @@ -694,7 +804,8 @@

    Unit Testing Domain Models

    More Types for More Type Hints
    -

    If you really want to go to town with type hints, you could go so far as +

    +If you really want to go to town with type hints, you could go so far as wrapping primitive types by using typing.NewType:

    @@ -729,9 +840,12 @@

    Unit Testing Domain Models

    -

    Dataclasses Are Great for Value Objects

    +

    Dataclasses Are Great for Value Objects

    -

    We’ve used line liberally in the previous code listings, but what is a +

    + + +We’ve used line liberally in the previous code listings, but what is a line? In our business language, an order has multiple line items, where each line has a SKU and a quantity. We can imagine that a simple YAML file containing order information might look like this:

    @@ -741,14 +855,14 @@

    Dataclasses Are Great for Valu
    @@ -759,7 +873,8 @@

    Dataclasses Are Great for Valu it’s not something that uniquely identifies the line itself.)

    -

    Whenever we have a business concept that has data but no identity, we +

    +Whenever we have a business concept that has data but no identity, we often choose to represent it using the Value Object pattern. A value object is any domain object that is uniquely identified by the data it holds; we usually make them immutable:

    @@ -769,7 +884,7 @@

    Dataclasses Are Great for Valu
    -

    One of the nice things that dataclasses (or namedtuples) give us is value +

    +One of the nice things that dataclasses (or namedtuples) give us is value equality, which is the fancy way of saying, "Two lines with the same orderid, sku, and qty are equal."

    @@ -792,7 +908,7 @@

    Dataclasses Are Great for Valu from typing import NamedTuple from collections import namedtuple -@dataclass(frozen=True) +@dataclass(frozen=True) class Name: first_name: str surname: str @@ -812,7 +928,8 @@

    Dataclasses Are Great for Valu

    -

    These value objects match our real-world intuition about how their values +

    +These value objects match our real-world intuition about how their values work. It doesn’t matter which £10 note we’re talking about, because they all have the same value. Likewise, two names are equal if both the first and last names match; and two lines are equivalent if they have the same customer order, @@ -820,8 +937,8 @@

    Dataclasses Are Great for Valu object, though. In fact, it’s common to support operations on values; for example, mathematical operators:

    -
    -
    Math with value objects
    +
    +
    Testing Math with value objects
    +
    +

    + +To get those tests to actually pass you’ll need to start implementing some +magic methods on our Money class:

    +
    +
    +
    Implementing Math with value objects
    +
    + +
    +
    -

    Value Objects and Entities

    +

    Value Objects and Entities

    -

    An order line is uniquely identified by its order ID, SKU, and quantity; if we +

    + +An order line is uniquely identified by its order ID, SKU, and quantity; if we change one of those values, we now have a new line. That’s the definition of a value object: any object that is identified only by its data and doesn’t have a long-lived identity. What about a batch, though? That is identified by a reference.

    -

    We use the term entity to describe a domain object that has long-lived +

    +We use the term entity to describe a domain object that has long-lived identity. On the previous page, we introduced a Name class as a value object. If we take the name Harry Percival and change one letter, we have the new Name object Barry Percival.

    @@ -907,13 +1051,16 @@

    Value Objects and Entities

    -

    Entities, unlike values, have identity equality. We can change their values, +

    + +Entities, unlike values, have identity equality. We can change their values, and they are still recognizably the same thing. Batches, in our example, are entities. We can allocate lines to a batch, or change the date that we expect it to arrive, and it will still be the same entity.

    -

    We usually make this explicit in code by implementing equality operators on +

    +We usually make this explicit in code by implementing equality operators on entities:

    @@ -926,7 +1073,7 @@

    Value Objects and Entities

    def __eq__(self, other): if not isinstance(other, Batch): - return False + return False return other.reference == self.reference def __hash__(self): @@ -936,11 +1083,15 @@

    Value Objects and Entities

    -

    Python’s __eq__ magic method +

    + +Python’s __eq__ magic method defines the behavior of the class for the == operator.[5]

    -

    For both entity and value objects, it’s also worth thinking through how +

    + +For both entity and value objects, it’s also worth thinking through how __hash__ will work. It’s the magic method Python uses to control the behavior of objects when you add them to sets or use them as dict keys; you can find more info in the Python docs.

    @@ -965,10 +1116,13 @@

    Value Objects and Entities

    Warning
    -This is tricky territory; you shouldn’t modify __hash__ without - also modifying __eq__. If you’re not sure what you’re doing, - further reading is suggested. - "Python Hashes and Equality" by our tech reviewer Hynek Schlawack is a good place to start. +This is tricky territory; you shouldn’t modify __hash__ + without also modifying __eq__. If you’re not sure what + you’re doing, further reading is suggested. + "Python Hashes and Equality" by our tech reviewer + Hynek Schlawack is a good place to start. + + @@ -976,9 +1130,11 @@

    Value Objects and Entities

    -

    Not Everything Has to Be an Object: A Domain Service Function

    +

    Not Everything Has to Be an Object: A Domain Service Function

    -

    We’ve made a model to represent batches, but what we actually need +

    + +We’ve made a model to represent batches, but what we actually need to do is allocate order lines against a specific set of batches that represent all our stock.

    @@ -994,11 +1150,14 @@

    Not Every

    -

    Evans discusses the idea of Domain Service -operations that don’t have a natural home in an entity or value object.[6] A +

    +Evans discusses the idea of Domain Service +operations that don’t have a natural home in an entity or value +object.[6] A thing that allocates an order line, given a set of batches, sounds a lot like a function, and we can take advantage of the fact that Python is a multiparadigm -language and just make it a function.

    +language and just make it a function. +

    Let’s see how we might test-drive such a function:

    @@ -1009,7 +1168,7 @@

    Not Every
    def test_prefers_current_stock_batches_to_shipments():
    -    in_stock_batch = Batch("in-stock-batch", "RETRO-CLOCK", 100, eta=None)
    +    in_stock_batch = Batch("in-stock-batch", "RETRO-CLOCK", 100, eta=None)
         shipment_batch = Batch("shipment-batch", "RETRO-CLOCK", 100, eta=tomorrow)
         line = OrderLine("oref", "RETRO-CLOCK", 10)
     
    @@ -1033,7 +1192,7 @@ 

    Not Every def test_returns_allocated_batch_ref(): - in_stock_batch = Batch("in-stock-batch-ref", "HIGHBROW-POSTER", 100, eta=None) + in_stock_batch = Batch("in-stock-batch-ref", "HIGHBROW-POSTER", 100, eta=None) shipment_batch = Batch("shipment-batch-ref", "HIGHBROW-POSTER", 100, eta=tomorrow) line = OrderLine("oref", "HIGHBROW-POSTER", 10) allocation = allocate(line, [in_stock_batch, shipment_batch]) @@ -1043,7 +1202,8 @@

    Not Every

    -

    And our service might look like this:

    +

    +And our service might look like this:

    A standalone function for our domain service (model.py)
    @@ -1051,9 +1211,7 @@

    Not Every
    def allocate(line: OrderLine, batches: List[Batch]) -> str:
    -    batch = next(
    -        b for b in sorted(batches) if b.can_allocate(line)
    -    )
    +    batch = next(b for b in sorted(batches) if b.can_allocate(line))
         batch.allocate(line)
         return batch.reference
    @@ -1061,9 +1219,11 @@

    Not Every

    -

    Python’s Magic Methods Let Us Use Our Models with Idiomatic Python

    +

    Python’s Magic Methods Let Us Use Our Models with Idiomatic Python

    -

    You may or may not like the use of next() in the preceding code, but we’re pretty +

    + +You may or may not like the use of next() in the preceding code, but we’re pretty sure you’ll agree that being able to use sorted() on our list of batches is nice, idiomatic Python.

    @@ -1079,10 +1239,10 @@

    Pyth ... def __gt__(self, other): - if self.eta is None: - return False - if other.eta is None: - return True + if self.eta is None: + return False + if other.eta is None: + return True return self.eta > other.eta

    @@ -1093,13 +1253,14 @@

    Pyth

    -

    Exceptions Can Express Domain Concepts Too

    +

    Exceptions Can Express Domain Concepts Too

    -

    We have one final concept to cover: exceptions -can be used to express domain concepts too. In our conversations -with domain experts, we’ve learned about the possibility that -an order cannot be allocated because we are out of stock, and -we can capture that by using a domain exception:

    +

    + +We have one final concept to cover: exceptions can be used to express domain +concepts too. In our conversations with domain experts, we’ve learned about the +possibility that an order cannot be allocated because we are out of stock, +and we can capture that by using a domain exception:

    Testing out-of-stock exception (test_allocate.py)
    @@ -1107,11 +1268,11 @@

    Exceptions Can Express Doma
    def test_raises_out_of_stock_exception_if_cannot_allocate():
    -    batch = Batch('batch1', 'SMALL-FORK', 10, eta=today)
    -    allocate(OrderLine('order1', 'SMALL-FORK', 10), [batch])
    +    batch = Batch("batch1", "SMALL-FORK", 10, eta=today)
    +    allocate(OrderLine("order1", "SMALL-FORK", 10), [batch])
     
    -    with pytest.raises(OutOfStock, match='SMALL-FORK'):
    -        allocate(OrderLine('order2', 'SMALL-FORK', 1), [batch])
    + with pytest.raises(OutOfStock, match="SMALL-FORK"): + allocate(OrderLine("order2", "SMALL-FORK", 1), [batch])

    @@ -1125,7 +1286,8 @@

    Exceptions Can Express Doma

    This is the part of your code that is closest to the business, the most likely to change, and the place where you deliver the -most value to the business. Make it easy to understand and modify.

    +most value to the business. Make it easy to understand and modify. +

    Distinguish entities from value objects
    @@ -1134,19 +1296,23 @@

    Exceptions Can Express Doma a Value Object, it represents a different object. In contrast, an entity has attributes that may vary over time and it will still be the same entity. It’s important to define what does uniquely identify -an entity (usually some sort of name or reference field).

    +an entity (usually some sort of name or reference field). + +

    Not everything has to be an object

    Python is a multiparadigm language, so let the "verbs" in your code be functions. For every FooManager, BarBuilder, or BazFactory, there’s often a more expressive and readable manage_foo(), build_bar(), -or get_baz() waiting to happen.

    +or get_baz() waiting to happen. +

    This is the time to apply your best OO design principles

    Revisit the SOLID principles and all the other good heuristics like "has a versus is-a," -"prefer composition over inheritance," and so on.

    +"prefer composition over inheritance," and so on. +

    You’ll also want to think about consistency boundaries and aggregates
    @@ -1175,7 +1341,7 @@

    Exceptions Can Express Doma batch = next( ... except StopIteration: - raise OutOfStock(f'Out of stock for sku {line.sku}') + raise OutOfStock(f"Out of stock for sku {line.sku}")

    @@ -1190,13 +1356,19 @@

    Exceptions Can Express Doma
    Figure 4. Our domain model at the end of the chapter
    -

    That’ll probably do for now! We have a domain service that we can use for our +

    +That’ll probably do for now! We have a domain service that we can use for our first use case. But first we’ll need a database…​

    +

    @@ -1207,8 +1379,8 @@

    Exceptions Can Express Doma 2. In previous Python versions, we might have used a namedtuple. You could also check out Hynek Schlawack’s excellent attrs.

    -3. Or perhaps you think there’s not enough code? What about some sort of check that the SKU in the OrderLine matches Batch.sku? We saved some thoughts on validation for [appendix_validation -
    +3. Or perhaps you think there’s not enough code? What about some sort of check that the SKU in the OrderLine matches Batch.sku? We saved some thoughts on validation for [appendix_validation]. +
    4. It is appalling. Please, please don’t do this. —Harry
    @@ -1221,93 +1393,22 @@

    Exceptions Can Express Doma -
    diff --git a/docs/book/chapter_02_repository.html b/book/chapter_02_repository.html similarity index 86% rename from docs/book/chapter_02_repository.html rename to book/chapter_02_repository.html index 7b027f9..9b77eef 100644 --- a/docs/book/chapter_02_repository.html +++ b/book/chapter_02_repository.html @@ -3,7 +3,7 @@ - + Repository Pattern + @@ -81,7 +183,7 @@
    -

    Repository Pattern

    +

    2: Repository Pattern

    It’s time to make good on our promise to use the dependency inversion principle as a way of decoupling our core logic from infrastructural concerns.

    -

    We’ll introduce the Repository pattern, a simplifying abstraction over data storage, +

    + + +We’ll introduce the Repository pattern, a simplifying abstraction over data storage, allowing us to decouple our model layer from the data layer. We’ll present a concrete example of how this simplifying abstraction makes our system more testable by hiding the complexities of the database.

    @@ -154,9 +259,10 @@

    Repository Pattern

    -

    Persisting Our Domain Model

    +

    Persisting Our Domain Model

    -

    In [chapter_01_domain_model] we built a simple domain model that can allocate orders +

    +In [chapter_01_domain_model] we built a simple domain model that can allocate orders to batches of stock. It’s easy for us to write tests against this code because there aren’t any dependencies or infrastructure to set up. If we needed to run a database or an API and create test data, our tests would be harder to write @@ -169,7 +275,8 @@

    Persisting Our Domain Model

    how we can connect our idealized domain model to external state.

    -

    We expect to be working in an agile manner, so our priority is to get to a +

    +We expect to be working in an agile manner, so our priority is to get to a minimum viable product as quickly as possible. In our case, that’s going to be a web API. In a real project, you might dive straight in with some end-to-end tests and start plugging in a web framework, test-driving things outside-in.

    @@ -181,7 +288,7 @@

    Persisting Our Domain Model

    -

    Some Pseudocode: What Are We Going to Need?

    +

    Some Pseudocode: What Are We Going to Need?

    When we build our first API endpoint, we know we’re going to have some code that looks more or less like the following.

    @@ -191,7 +298,7 @@

    Some Pseudocode: What Are We
    -

    Applying the DIP to Data Access

    +

    Applying the DIP to Data Access

    -

    As mentioned in the introduction, a layered architecture is a common +

    + +As mentioned in the introduction, a layered architecture is a common approach to structuring a system that has a UI, some logic, and a database (see Layered architecture).

    @@ -248,12 +358,14 @@

    Applying the DIP to Data Access

    below it.

    -

    But we want our domain model to have no dependencies whatsoever.[1] +

    +But we want our domain model to have no dependencies whatsoever.[1] We don’t want infrastructure concerns bleeding over into our domain model and slowing our unit tests or our ability to make changes.

    -

    Instead, as discussed in the introduction, we’ll think of our model as being on the +

    +Instead, as discussed in the introduction, we’ll think of our model as being on the "inside," and dependencies flowing inward to it; this is what people sometimes call onion architecture (see Onion architecture).

    @@ -296,7 +408,9 @@

    Applying the DIP to Data Access

    -

    Although some people like to nitpick over the differences, all these are +

    + +Although some people like to nitpick over the differences, all these are pretty much names for the same thing, and they all boil down to the dependency inversion principle: high-level modules (the domain) should not depend on low-level ones (the infrastructure).[2]

    @@ -310,9 +424,10 @@

    Applying the DIP to Data Access

    -

    Reminder: Our Model

    +

    Reminder: Our Model

    -

    Let’s remind ourselves of our domain model (see Our model): +

    +Let’s remind ourselves of our domain model (see Our model): an allocation is the concept of linking an OrderLine to a Batch. We’re storing the allocations as a collection on our Batch object.

    @@ -326,25 +441,31 @@

    Reminder: Our Model

    Let’s see how we might translate this to a relational database.

    -

    The "Normal" ORM Way: Model Depends on ORM

    +

    The "Normal" ORM Way: Model Depends on ORM

    -

    These days, it’s unlikely that your team members are hand-rolling their own SQL queries. +

    + +These days, it’s unlikely that your team members are hand-rolling their own SQL queries. Instead, you’re almost certainly using some kind of framework to generate SQL for you based on your model objects.

    -

    These frameworks are called object-relational mappers (ORMs) because they exist to +

    +These frameworks are called object-relational mappers (ORMs) because they exist to bridge the conceptual gap between the world of objects and domain modeling and the world of databases and relational algebra.

    -

    The most important thing an ORM gives us is persistence ignorance: the idea +

    +The most important thing an ORM gives us is persistence ignorance: the idea that our fancy domain model doesn’t need to know anything about how data is loaded or persisted. This helps keep our domain clean of direct dependencies on particular database technologies.[3]

    -

    But if you follow the typical SQLAlchemy tutorial, you’ll end up with something +

    + +But if you follow the typical SQLAlchemy tutorial, you’ll end up with something like this:

    @@ -359,10 +480,10 @@

    The "Normal" ORM Way: Model De Base = declarative_base() class Order(Base): - id = Column(Integer, primary_key=True) + id = Column(Integer, primary_key=True) class OrderLine(Base): - id = Column(Integer, primary_key=True) + id = Column(Integer, primary_key=True) sku = Column(String(250)) qty = Integer(String(250)) order_id = Column(Integer, ForeignKey('order.id')) @@ -385,7 +506,9 @@

    The "Normal" ORM Way: Model De
    Django’s ORM Is Essentially the Same, but More Restrictive
    -

    If you’re more used to Django, the preceding "declarative" SQLAlchemy snippet +

    + +If you’re more used to Django, the preceding "declarative" SQLAlchemy snippet translates to something like this:

    @@ -421,9 +544,15 @@

    The "Normal" ORM Way: Model De

    -

    Inverting the Dependency: ORM Depends on Model

    +

    Inverting the Dependency: ORM Depends on Model

    -

    Well, thankfully, that’s not the only way to use SQLAlchemy. The alternative is +

    + + + + + +Well, thankfully, that’s not the only way to use SQLAlchemy. The alternative is to define your schema separately, and to define an explicit mapper for how to convert between the schema and our domain model, what SQLAlchemy calls a classical mapping:

    @@ -441,11 +570,12 @@

    Inverting the Dependency metadata = MetaData() order_lines = Table( #(2) - 'order_lines', metadata, - Column('id', Integer, primary_key=True, autoincrement=True), - Column('sku', String(255)), - Column('qty', Integer, nullable=False), - Column('orderid', String(255)), + "order_lines", + metadata, + Column("id", Integer, primary_key=True, autoincrement=True), + Column("sku", String(255)), + Column("qty", Integer, nullable=False), + Column("orderid", String(255)), ) ... @@ -484,7 +614,8 @@

    Inverting the Dependency domain classes, as we’ll see.

    -

    When you’re first trying to build your ORM config, it can be useful to write +

    +When you’re first trying to build your ORM config, it can be useful to write tests for it, as in the following example:

    @@ -494,7 +625,7 @@

    Inverting the Dependency
    def test_orderline_mapper_can_load_lines(session):  #(1)
         session.execute(
    -        'INSERT INTO order_lines (orderid, sku, qty) VALUES '
    +        "INSERT INTO order_lines (orderid, sku, qty) VALUES "
             '("order1", "RED-CHAIR", 12),'
             '("order1", "RED-TABLE", 13),'
             '("order2", "BLUE-LIPSTICK", 14)'
    @@ -526,7 +657,8 @@ 

    Inverting the Dependency fixtures for the purposes of this book, but the short explanation is that you can define common dependencies for your tests as "fixtures," and pytest will inject them to the tests that need them by looking at their -function arguments. In this case, it’s a SQLAlchemy database session.

    +function arguments. In this case, it’s a SQLAlchemy database session. +

    @@ -553,7 +685,8 @@

    Inverting the Dependency Zen of Python says, "Practicality beats purity!"

    -

    At this point, though, our API endpoint might look something like +

    +At this point, though, our API endpoint might look something like the following, and we could get it to work just fine:

    @@ -561,7 +694,7 @@

    Inverting the Dependency
    -

    Introducing the Repository Pattern

    +

    Introducing the Repository Pattern

    -

    The Repository pattern is an abstraction over persistent storage. It hides the +

    + +The Repository pattern is an abstraction over persistent storage. It hides the boring details of data access by pretending that all of our data is in memory.

    @@ -624,16 +759,19 @@

    Introducing the Repository Pattern< .save() method; we just fetch the object we care about and modify it in memory.

    -

    The Repository in the Abstract

    +

    The Repository in the Abstract

    -

    The simplest repository has just two methods: add() to put a new item in the -repository, and get() to return a previously added item.[6].] +

    + +The simplest repository has just two methods: add() to put a new item in the +repository, and get() to return a previously added item.[6] We stick rigidly to using these methods for data access in our domain and our service layer. This self-imposed simplicity stops us from coupling our domain model to the database.

    -

    Here’s what an abstract base class (ABC) for our repository would look like:

    +

    +Here’s what an abstract base class (ABC) for our repository would look like:

    The simplest possible repository (repository.py)
    @@ -641,12 +779,11 @@

    The Repository in the Abstract

    class AbstractRepository(abc.ABC):
    -
    -    @abc.abstractmethod  #(1)
    +    @abc.abstractmethod  #(1)
         def add(self, batch: model.Batch):
             raise NotImplementedError  #(2)
     
    -    @abc.abstractmethod
    +    @abc.abstractmethod
         def get(self, reference) -> model.Batch:
             raise NotImplementedError
    @@ -657,12 +794,15 @@

    The Repository in the Abstract

    1. Python tip: @abc.abstractmethod is one of the only things that makes - ABCs actually "work" in Python. Python will refuse to let you instantiate - a class that does not implement all the abstractmethods defined in its - parent class.[7]

      +ABCs actually "work" in Python. Python will refuse to let you instantiate +a class that does not implement all the abstractmethods defined in its +parent class.[7] + +

    2. -

      raise NotImplementedError is nice, but it’s neither necessary nor sufficient. In fact, your abstract methods can have real behavior that subclasses +

      raise NotImplementedError is nice, but it’s neither necessary nor sufficient. +In fact, your abstract methods can have real behavior that subclasses can call out to, if you really want.

    @@ -671,26 +811,30 @@

    The Repository in the Abstract

    Abstract Base Classes, Duck Typing, and Protocols
    -

    We’re using abstract base classes in this book for didactic reasons: we hope +

    + +We’re using abstract base classes in this book for didactic reasons: we hope they help explain what the interface of the repository abstraction is.

    -

    In real life, we’ve sometimes found ourselves deleting ABCs from our production +

    +In real life, we’ve sometimes found ourselves deleting ABCs from our production code, because Python makes it too easy to ignore them, and they end up unmaintained and, at worst, misleading. In practice we often just rely on Python’s duck typing to enable abstractions. To a Pythonista, a repository is any object that has add(thing) and get(id) methods.

    -

    An alternative to look into is PEP -544 protocols. These give you typing without the possibility of inheritance, -which "prefer composition over inheritance" fans will particularly like.

    +

    +An alternative to look into is PEP 544 protocols. +These give you typing without the possibility of inheritance, which "prefer +composition over inheritance" fans will particularly like.

    -

    What Is the Trade-Off?

    +

    What Is the Trade-Off?

    @@ -704,7 +848,8 @@

    What Is the Trade-Off?

    -

    Whenever we introduce an architectural pattern in this book, we’ll always +

    +Whenever we introduce an architectural pattern in this book, we’ll always ask, "What do we get for this? And what does it cost us?"

    @@ -728,7 +873,8 @@

    What Is the Trade-Off?

    [appendix_csvs]), and as we’ll see, it is easy to fake out for unit tests.

    -

    In addition, the Repository pattern is so common in the DDD world that, if you +

    +In addition, the Repository pattern is so common in the DDD world that, if you do collaborate with programmers who have come to Python from the Java and C# worlds, they’re likely to recognize it. Repository pattern illustrates the pattern.

    @@ -761,7 +907,9 @@

    What Is the Trade-Off?

    -

    As always, we start with a test. This would probably be classified as an +

    + +As always, we start with a test. This would probably be classified as an integration test, since we’re checking that our code (the repository) is correctly integrated with the database; hence, the tests tend to mix raw SQL with calls and assertions on our own code.

    @@ -786,16 +934,16 @@

    What Is the Trade-Off?

    def test_repository_can_save_a_batch(session):
    -    batch = model.Batch("batch1", "RUSTY-SOAPDISH", 100, eta=None)
    +    batch = model.Batch("batch1", "RUSTY-SOAPDISH", 100, eta=None)
     
         repo = repository.SqlAlchemyRepository(session)
         repo.add(batch)  #(1)
         session.commit()  #(2)
     
    -    rows = list(session.execute(
    -        'SELECT reference, sku, _purchased_quantity, eta FROM "batches"'  #(3)
    -    ))
    -    assert rows == [("batch1", "RUSTY-SOAPDISH", 100, None)]
    + rows = session.execute( #(3) + 'SELECT reference, sku, _purchased_quantity, eta FROM "batches"' + ) + assert list(rows) == [("batch1", "RUSTY-SOAPDISH", 100, None)]
    @@ -817,7 +965,9 @@

    What Is the Trade-Off?

    -

    The next test involves retrieving batches and allocations, so it’s more +

    + +The next test involves retrieving batches and allocations, so it’s more complex:

    @@ -827,15 +977,16 @@

    What Is the Trade-Off?

    def insert_order_line(session):
         session.execute(  #(1)
    -        'INSERT INTO order_lines (orderid, sku, qty)'
    +        "INSERT INTO order_lines (orderid, sku, qty)"
             ' VALUES ("order1", "GENERIC-SOFA", 12)'
         )
         [[orderline_id]] = session.execute(
    -        'SELECT id FROM order_lines WHERE orderid=:orderid AND sku=:sku',
    -        dict(orderid="order1", sku="GENERIC-SOFA")
    +        "SELECT id FROM order_lines WHERE orderid=:orderid AND sku=:sku",
    +        dict(orderid="order1", sku="GENERIC-SOFA"),
         )
         return orderline_id
     
    +
     def insert_batch(session, batch_id):  #(2)
         ...
     
    @@ -843,12 +994,12 @@ 

    What Is the Trade-Off?

    orderline_id = insert_order_line(session) batch1_id = insert_batch(session, "batch1") insert_batch(session, "batch2") - insert_allocation(session, orderline_id, batch1_id) #(3) + insert_allocation(session, orderline_id, batch1_id) #(2) repo = repository.SqlAlchemyRepository(session) retrieved = repo.get("batch1") - expected = model.Batch("batch1", "GENERIC-SOFA", 100, eta=None) + expected = model.Batch("batch1", "GENERIC-SOFA", 100, eta=None) assert retrieved == expected # Batch.__eq__ only compares reference #(3) assert retrieved.sku == expected.sku #(4) assert retrieved._purchased_quantity == expected._purchased_quantity @@ -882,7 +1033,8 @@

    What Is the Trade-Off?

    -

    Whether or not you painstakingly write tests for every model is a judgment +

    +Whether or not you painstakingly write tests for every model is a judgment call. Once you have one class tested for create/modify/save, you might be happy to go on and do the others with a minimal round-trip test, or even nothing at all, if they all follow a similar pattern. In our case, the ORM config @@ -898,7 +1050,6 @@

    What Is the Trade-Off?

    class SqlAlchemyRepository(AbstractRepository):
    -
         def __init__(self, session):
             self.session = session
     
    @@ -915,14 +1066,17 @@ 

    What Is the Trade-Off?

    -

    And now our Flask endpoint might look something like the following:

    +

    + + +And now our Flask endpoint might look something like the following:

    Using our repository directly in our API endpoint
    +

    @@ -1209,8 +1386,8 @@

    Wrap-Up

    5. Shout-out to the amazingly helpful SQLAlchemy maintainers, and to Mike Bayer in particular.
    -6. You may be thinking, "What about list or delete or update?" However, in an ideal world, we modify our model objects one at a time, and delete is usually handled as a soft-delete—i.e., batch.cancel(). Finally, update is taken care of by the Unit of Work pattern, as you’ll see in [chapter_06_uow -
    +6. You may be thinking, "What about list or delete or update?" However, in an ideal world, we modify our model objects one at a time, and delete is usually handled as a soft-delete—i.e., batch.cancel(). Finally, update is taken care of by the Unit of Work pattern, as you’ll see in [chapter_06_uow]. +
    7. To really reap the benefits of ABCs (such as they may be), be running helpers like pylint and mypy.
    @@ -1220,93 +1397,22 @@

    Wrap-Up

    -
    diff --git a/docs/book/chapter_03_abstractions.html b/book/chapter_03_abstractions.html similarity index 69% rename from docs/book/chapter_03_abstractions.html rename to book/chapter_03_abstractions.html index 5572a44..eab4826 100644 --- a/docs/book/chapter_03_abstractions.html +++ b/book/chapter_03_abstractions.html @@ -3,7 +3,7 @@ - + A Brief Interlude: On Coupling and Abstractions + @@ -81,7 +183,7 @@
    -

    A Brief Interlude: On Coupling and Abstractions

    +

    3: A Brief Interlude: On Coupling and Abstractions

    -

    Allow us a brief digression on the subject of abstractions, dear reader. +

    +Allow us a brief digression on the subject of abstractions, dear reader. We’ve talked about abstractions quite a lot. The Repository pattern is an abstraction over permanent storage, for example. But what makes a good abstraction? What do we want from abstractions? And how do they relate to testing?

    @@ -137,7 +240,8 @@

    A Brief Interlude: On Coupling -

    A key theme in this book, hidden among the fancy patterns, is that we can use +

    +A key theme in this book, hidden among the fancy patterns, is that we can use simple abstractions to hide messy details. When we’re writing code for fun, or in a kata,[1] we get to play with ideas freely, hammering things out and refactoring @@ -145,14 +249,18 @@

    A Brief Interlude: On Coupling -

    When we’re unable to change component A for fear of breaking component B, we say +

    + +When we’re unable to change component A for fear of breaking component B, we say that the components have become coupled. Locally, coupling is a good thing: it’s a sign that our code is working together, each component supporting the others, all of them fitting in place like the gears of a watch. In jargon, we say this works when there is high cohesion between the coupled elements.

    -

    Globally, coupling is a nuisance: it increases the risk and the cost of changing +

    + +Globally, coupling is a nuisance: it increases the risk and the cost of changing our code, sometimes to the point where we feel unable to make any changes at all. This is the problem with the Ball of Mud pattern: as the application grows, if we’re unable to prevent coupling between elements that have no cohesion, that @@ -160,7 +268,9 @@

    A Brief Interlude: On Coupling -

    We can reduce the degree of coupling within a system +

    + +We can reduce the degree of coupling within a system (Lots of coupling) by abstracting away the details (Less coupling).

    @@ -215,9 +325,13 @@

    A Brief Interlude: On Coupling -

    Abstracting State Aids Testability

    +

    Abstracting State Aids Testability

    -

    Let’s see an example. Imagine we want to write code for synchronizing two +

    + + + +Let’s see an example. Imagine we want to write code for synchronizing two file directories, which we’ll call the source and the destination:

    @@ -235,7 +349,8 @@

    Abstracting State Aids Testability<

    -

    Our first and third requirements are simple enough: we can just compare two +

    +Our first and third requirements are simple enough: we can just compare two lists of paths. Our second is trickier, though. To detect renames, we’ll have to inspect the content of files. For this, we can use a hashing function like MD5 or SHA-1. The code to generate a SHA-1 hash from a file is simple @@ -248,13 +363,14 @@

    Abstracting State Aids Testability<
    BLOCKSIZE = 65536
     
    +
     def hash_file(path):
         hasher = hashlib.sha1()
    -    with path.open("rb") as file:
    -        buf = file.read(BLOCKSIZE)
    +    with path.open("rb") as file:
    +        buf = file.read(BLOCKSIZE)
             while buf:
                 hasher.update(buf)
    -            buf = file.read(BLOCKSIZE)
    +            buf = file.read(BLOCKSIZE)
         return hasher.hexdigest()

    @@ -284,6 +400,7 @@

    Abstracting State Aids Testability< import shutil from pathlib import Path + def sync(source, dest): # Walk the source folder and build a dict of filenames and their hashes source_hashes = {} @@ -311,8 +428,8 @@

    Abstracting State Aids Testability< # for every file that appears in source but not target, copy the file to # the target - for src_hash, fn in source_hashes.items(): - if src_hash not in seen: + for source_hash, fn in source_hashes.items(): + if source_hash not in seen: shutil.copy(Path(source) / fn, Path(dest) / fn)

    @@ -333,11 +450,11 @@

    Abstracting State Aids Testability< dest = tempfile.mkdtemp() content = "I am a very useful file" - (Path(source) / 'my-file').write_text(content) + (Path(source) / "my-file").write_text(content) sync(source, dest) - expected_path = Path(dest) / 'my-file' + expected_path = Path(dest) / "my-file" assert expected_path.exists() assert expected_path.read_text() == content @@ -352,18 +469,17 @@

    Abstracting State Aids Testability< dest = tempfile.mkdtemp() content = "I am a file that was renamed" - source_path = Path(source) / 'source-filename' - old_dest_path = Path(dest) / 'dest-filename' - expected_dest_path = Path(dest) / 'source-filename' + source_path = Path(source) / "source-filename" + old_dest_path = Path(dest) / "dest-filename" + expected_dest_path = Path(dest) / "source-filename" source_path.write_text(content) old_dest_path.write_text(content) sync(source, dest) - assert old_dest_path.exists() is False + assert old_dest_path.exists() is False assert expected_dest_path.read_text() == content - finally: shutil.rmtree(source) shutil.rmtree(dest) @@ -372,7 +488,9 @@

    Abstracting State Aids Testability<

    -

    Wowsers, that’s a lot of setup for two simple cases! The problem is that +

    + +Wowsers, that’s a lot of setup for two simple cases! The problem is that our domain logic, "figure out the difference between two directories," is tightly coupled to the I/O code. We can’t run our difference algorithm without calling the pathlib, shutil, and hashlib modules.

    @@ -391,7 +509,12 @@

    Abstracting State Aids Testability< or to cloud storage?

    -

    Our high-level code is coupled to low-level details, and it’s making life hard. +

    + + + + +Our high-level code is coupled to low-level details, and it’s making life hard. As the scenarios we consider get more complex, our tests will get more unwieldy. We can definitely refactor these tests (some of the cleanup could go into pytest fixtures, for example) but as long as we’re doing filesystem operations, they’re @@ -399,12 +522,15 @@

    Abstracting State Aids Testability<

    -

    Choosing the Right Abstraction(s)

    +

    Choosing the Right Abstraction(s)

    -

    What could we do to rewrite our code to make it more testable?

    +

    + +What could we do to rewrite our code to make it more testable?

    -

    First, we need to think about what our code needs from the filesystem. +

    +First, we need to think about what our code needs from the filesystem. Reading through the code, we can see that three distinct things are happening. We can think of these as three distinct responsibilities that the code has:

    @@ -424,7 +550,8 @@

    Choosing the Right Abstraction(s)

    -

    Remember that we want to find simplifying abstractions for each of these +

    +Remember that we want to find simplifying abstractions for each of these responsibilities. That will let us hide the messy details so we can focus on the interesting logic.[2]

    @@ -444,10 +571,13 @@

    Choosing the Right Abstraction(s)

    -

    For steps 1 and 2, we’ve already intuitively started using an abstraction, a -dictionary of hashes to paths. You may already have been thinking, "Why not build up a dictionary for the destination folder as well as the source, and -then we just compare two dicts?" That seems like a nice way to abstract -the current state of the filesystem:

    +

    + +For steps 1 and 2, we’ve already intuitively started using an abstraction, a +dictionary of hashes to paths. You may already have been thinking, "Why not +build up a dictionary for the destination folder as well as the source, and +then we just compare two dicts?" That seems like a nice way to abstract the +current state of the filesystem:

    @@ -460,7 +590,8 @@

    Choosing the Right Abstraction(s)

    actual move/copy/delete filesystem interaction?

    -

    We’ll apply a trick here that we’ll employ on a grand scale later in +

    +We’ll apply a trick here that we’ll employ on a grand scale later in the book. We’re going to separate what we want to do from how to do it. We’re going to make our program output a list of commands that look like this:

    @@ -471,7 +602,8 @@

    Choosing the Right Abstraction(s)

    -

    Now we could write tests that just use two filesystem dicts as inputs, and we would +

    +Now we could write tests that just use two filesystem dicts as inputs, and we would expect lists of tuples of strings representing actions as outputs.

    @@ -485,14 +617,14 @@

    Choosing the Right Abstraction(s)

    -

    Implementing Our Chosen Abstractions

    +

    Implementing Our Chosen Abstractions

    -

    That’s all very well, but how do we actually write those new +

    + + + +That’s all very well, but how do we actually write those new tests, and how do we change our implementation to make it all work?

    -

    Our goal is to isolate the clever part of our system, and to be able to test it +

    + + +Our goal is to isolate the clever part of our system, and to be able to test it thoroughly without needing to set up a real filesystem. We’ll create a "core" of code that has no dependencies on external state and then see how it responds when we give it input from the outside world (this kind of approach was characterized @@ -516,7 +655,10 @@

    Implementing Our Chosen Abstracti Core, Imperative Shell, or FCIS).

    -

    Let’s start off by splitting the code to separate the stateful parts from +

    + + +Let’s start off by splitting the code to separate the stateful parts from the logic.

    @@ -538,11 +680,11 @@

    Implementing Our Chosen Abstracti # imperative shell step 3, apply outputs for action, *paths in actions: - if action == 'copy': + if action == "COPY": shutil.copyfile(*paths) - if action == 'move': + if action == "MOVE": shutil.move(*paths) - if action == 'delete': + if action == "DELETE": os.remove(paths[0])

    @@ -555,12 +697,13 @@

    Implementing Our Chosen Abstracti isolates the I/O part of our application.

  • -

    Here is where carve out the functional core, the business logic.

    +

    Here is where we carve out the functional core, the business logic.

  • -

    The code to build up the dictionary of paths and hashes is now trivially easy +

    +The code to build up the dictionary of paths and hashes is now trivially easy to write:

    @@ -589,21 +732,21 @@

    Implementing Our Chosen Abstracti
    -
    def determine_actions(src_hashes, dst_hashes, src_folder, dst_folder):
    -    for sha, filename in src_hashes.items():
    -        if sha not in dst_hashes:
    -            sourcepath = Path(src_folder) / filename
    -            destpath = Path(dst_folder) / filename
    -            yield 'copy', sourcepath, destpath
    +
    def determine_actions(source_hashes, dest_hashes, source_folder, dest_folder):
    +    for sha, filename in source_hashes.items():
    +        if sha not in dest_hashes:
    +            sourcepath = Path(source_folder) / filename
    +            destpath = Path(dest_folder) / filename
    +            yield "COPY", sourcepath, destpath
     
    -        elif dst_hashes[sha] != filename:
    -            olddestpath = Path(dst_folder) / dst_hashes[sha]
    -            newdestpath = Path(dst_folder) / filename
    -            yield 'move', olddestpath, newdestpath
    +        elif dest_hashes[sha] != filename:
    +            olddestpath = Path(dest_folder) / dest_hashes[sha]
    +            newdestpath = Path(dest_folder) / filename
    +            yield "MOVE", olddestpath, newdestpath
     
    -    for sha, filename in dst_hashes.items():
    -        if sha not in src_hashes:
    -            yield 'delete', dst_folder / filename
    + for sha, filename in dest_hashes.items(): + if sha not in source_hashes: + yield "DELETE", dest_folder / filename
    @@ -617,16 +760,17 @@

    Implementing Our Chosen Abstracti
    def test_when_a_file_exists_in_the_source_but_not_the_destination():
    -    src_hashes = {'hash1': 'fn1'}
    -    dst_hashes = {}
    -    actions = determine_actions(src_hashes, dst_hashes, Path('/src'), Path('/dst'))
    -    assert list(actions) == [('copy', Path('/src/fn1'), Path('/dst/fn1'))]
    +    source_hashes = {"hash1": "fn1"}
    +    dest_hashes = {}
    +    actions = determine_actions(source_hashes, dest_hashes, Path("/src"), Path("/dst"))
    +    assert list(actions) == [("COPY", Path("/src/fn1"), Path("/dst/fn1"))]
    +
     
     def test_when_a_file_has_been_renamed_in_the_source():
    -    src_hashes = {'hash1': 'fn1'}
    -    dst_hashes = {'hash1': 'fn2'}
    -    actions = determine_actions(src_hashes, dst_hashes, Path('/src'), Path('/dst'))
    -    assert list(actions) == [('move', Path('/dst/fn2'), Path('/dst/fn1'))]
    + source_hashes = {"hash1": "fn1"} + dest_hashes = {"hash1": "fn2"} + actions = determine_actions(source_hashes, dest_hashes, Path("/src"), Path("/dst")) + assert list(actions) == [("MOVE", Path("/dst/fn2"), Path("/dst/fn1"))]

    @@ -636,7 +780,8 @@

    Implementing Our Chosen Abstracti changes—​from the low-level details of I/O, we can easily test the core of our code.

    -

    With this approach, we’ve switched from testing our main entrypoint function, +

    +With this approach, we’ve switched from testing our main entrypoint function, sync(), to testing a lower-level function, determine_actions(). You might decide that’s fine because sync() is now so simple. Or you might decide to keep some integration/acceptance tests to test that sync(). But there’s @@ -645,14 +790,18 @@

    Implementing Our Chosen Abstracti edge-to-edge testing.

    -

    Testing Edge to Edge with Fakes and Dependency Injection

    +

    Testing Edge to Edge with Fakes and Dependency Injection

    -

    When we start writing a new system, we often focus on the core logic first, +

    + + +When we start writing a new system, we often focus on the core logic first, driving it with direct unit tests. At some point, though, we want to test bigger chunks of the system together.

    -

    We could return to our end-to-end tests, but those are still as tricky to +

    +We could return to our end-to-end tests, but those are still as tricky to write and maintain as before. Instead, we often write tests that invoke a whole system together but fake the I/O, sort of edge to edge:

    @@ -661,25 +810,24 @@

    Testing Edge
    @@ -687,14 +835,14 @@

    Testing Edge
    1. -

      Our top-level function now exposes two new dependencies, a reader and a -filesystem.

      +

      Our top-level function now exposes a new dependency, a FileSystem.

    2. -

      We invoke the reader to produce our files dict.

      +

      We invoke filesystem.read() to produce our files dict.

    3. -

      We invoke the filesystem to apply the changes we detect.

      +

      We invoke the FileSystem's .copy(), .move() and .delete() methods +to apply the changes we detect.

    @@ -714,43 +862,56 @@

    Testing Edge

    -
    -
    Tests using DI
    +
    +

    The real (default) implementation of our FileSystem abstraction does real I/O:

    +
    +
    +
    The real dependency (sync.py)
    +
    +
    +
    +

    But the fake one is a wrapper around our chosen abstractions, +rather than doing real I/O:

    +
    +
    +
    Tests using DI
    +
    +
    @@ -758,17 +919,52 @@

    Testing Edge
    1. -

      Bob loves using lists to build simple test doubles, even though his -coworkers get mad. It means we can write tests like -assert 'foo' not in database.

      +

      We initialize our fake filesysem using the abstraction we chose to +represent filesystem state: dictionaries of hashes to paths.

    2. -

      Each method in our FakeFileSystem just appends something to the list so we -can inspect it later. This is an example of a spy object.

      +

      The action methods in our FakeFileSystem just appends a record to an list +of .actions so we can inspect it later. This means our test double is both +a "fake" and a "spy". + + +

    +

    So now our tests can act on the real, top-level sync() entrypoint, +but they do so using the FakeFilesystem(). In terms of their +setup and assertions, they end up looking quite similar to the ones +we wrote when testing directly against the functional core determine_actions() +function:

    +
    +
    +
    Tests using DI
    +
    + +
    +
    +

    The advantage of this approach is that our tests act on the exact same function that’s used by our production code. The disadvantage is that we have to make our stateful components explicit and pass them around. @@ -776,15 +972,23 @@

    Testing Edge as "test-induced design damage."

    -

    In either case, we can now work on fixing all the bugs in our implementation; +

    + + + +In either case, we can now work on fixing all the bugs in our implementation; enumerating tests for all the edge cases is now much easier.

    -

    Why Not Just Patch It Out?

    +

    Why Not Just Patch It Out?

    -

    At this point you may be scratching your head and thinking, -"Why don’t you just use mock.patch and save yourself the effort?""

    +

    + + + +At this point you may be scratching your head and thinking, +"Why don’t you just use mock.patch and save yourself the effort?"

    We avoid using mocks in this book and in our production code too. We’re not @@ -826,7 +1030,8 @@

    Why Not Just Patch It Out?

    Tests that use mocks tend to be more coupled to the implementation details of the codebase. That’s because mock tests verify the interactions between things: did we call shutil.copy with the right arguments? This coupling between -code and test tends to make tests more brittle, in our experience.

    +code and test tends to make tests more brittle, in our experience. +

  • Overuse of mocks leads to complicated test suites that fail to explain the @@ -852,7 +1057,10 @@

    Why Not Just Patch It Out?

    Mocks Versus Fakes; Classic-Style Versus London-School TDD
    -

    Here’s a short and somewhat simplistic definition of the difference between +

    + + +Here’s a short and somewhat simplistic definition of the difference between mocks and fakes:

    @@ -872,18 +1080,27 @@

    Why Not Just Patch It Out?

    -

    We’re slightly conflating mocks with spies and fakes with stubs here, and you +

    + + +We’re slightly conflating mocks with spies and fakes with stubs here, and you can read the long, correct answer in Martin Fowler’s classic essay on the subject called "Mocks Aren’t Stubs".

    -

    It also probably doesn’t help that the MagicMock objects provided by +

    + + +It also probably doesn’t help that the MagicMock objects provided by unittest.mock aren’t, strictly speaking, mocks; they’re spies, if anything. But they’re also often used as stubs or dummies. There, we promise we’re done with the test double terminology nitpicks now.

    -

    What about London-school versus classic-style TDD? You can read more about those +

    + + +What about London-school versus classic-style TDD? You can read more about those two in Martin Fowler’s article that we just cited, as well as on the Software Engineering Stack Exchange site, but in this book we’re pretty firmly in the classicist camp. We like to @@ -902,17 +1119,28 @@

    Why Not Just Patch It Out?

    when we return to the code after a long absence.

    -

    Tests that use too many mocks get overwhelmed with setup code that hides the +

    +Tests that use too many mocks get overwhelmed with setup code that hides the story we care about.

    -

    Steve Freeman has a great example of overmocked tests in his talk +

    + + + +Steve Freeman has a great example of overmocked tests in his talk "Test-Driven Development". You should also check out this PyCon talk, "Mocking and Patching Pitfalls", by our esteemed tech reviewer, Ed Jung, which also addresses mocking and its -alternatives. And while we’re recommending talks, don’t miss Brandon Rhodes talking about -"Hoisting Your I/O", -which really nicely covers the issues we’re talking about, using another simple example.

    +alternatives.

    +
    +
    +

    And while we’re recommending talks, check out the wonderful Brandon Rhodes +in "Hoisting Your I/O". It’s not actually about mocks, +but is instead about the general issue of decoupling business logic from I/O, +in which he uses a wonderfully simple illustrative example. + +

    @@ -927,6 +1155,8 @@

    Why Not Just Patch It Out?

    pyramid with as many unit tests as possible, and with the minimum number of E2E tests you need to feel confident. Read on to [types_of_test_rules_of_thumb] for more details. + +
    @@ -935,7 +1165,8 @@

    Why Not Just Patch It Out?

    So Which Do We Use In This Book? Functional or Object-Oriented Composition?
    -

    Both. Our domain model is entirely free of dependencies and side effects, +

    +Both. Our domain model is entirely free of dependencies and side effects, so that’s our functional core. The service layer that we build around it (in [chapter_04_service_layer]) allows us to drive the system edge to edge, and we use dependency injection to provide those services with stateful @@ -950,9 +1181,16 @@

    Why Not Just Patch It Out?

    -

    Wrap-Up

    +

    Wrap-Up

    -

    We’ll see this idea come up again and again in the book: we can make our +

    + + + + + + +We’ll see this idea come up again and again in the book: we can make our systems easier to test and maintain by simplifying the interface between our business logic and messy I/O. Finding the right abstraction is tricky, but here are a few heuristics and questions to ask yourself:

    @@ -965,13 +1203,19 @@

    Wrap-Up

    state?

  • -

    Where can I draw a line between my systems, where can I carve out a -seam -to stick that abstraction in?

    +

    Separate the what from the how: +can I use a data structure or DSL to represent the external effects I want to happen, +independently of how I plan to make them happen?

    +
  • +
  • +

    Where can I draw a line between my systems, +where can I carve out a seam +to stick that abstraction in? +

  • -

    What is a sensible way of dividing things into components with different -responsibilities? What implicit concepts can I make explicit?

    +

    What is a sensible way of dividing things into components with different responsibilities? +What implicit concepts can I make explicit?

  • What are the dependencies, and what is the core business logic?

    @@ -979,16 +1223,22 @@

    Wrap-Up

  • -

    Practice makes less imperfect! And now back to our regular programming…​

    +

    +Practice makes less imperfect! And now back to our regular programming…​

    +

    -1. A code kata is a small, contained programming challenge often used to practice TDD. See "Kata—The Only Way to Learn TDD" by Peter Provost. +1. A code kata is a small, contained programming challenge often used to practice TDD. See "Kata—The Only Way to Learn TDD" by Peter Provost.
    2. If you’re used to thinking in terms of interfaces, that’s what we’re trying to define here. @@ -999,93 +1249,22 @@

    Wrap-Up

    -
    diff --git a/docs/book/chapter_04_service_layer.html b/book/chapter_04_service_layer.html similarity index 75% rename from docs/book/chapter_04_service_layer.html rename to book/chapter_04_service_layer.html index cd2d4c0..2b79542 100644 --- a/docs/book/chapter_04_service_layer.html +++ b/book/chapter_04_service_layer.html @@ -3,7 +3,7 @@ - + Our First Use Case: Flask API and Service Layer + @@ -81,7 +183,7 @@
    -

    Our First Use Case: Flask API and Service Layer

    +

    4: Our First Use Case: Flask API and Service Layer

    -

    Back to our allocations project! Before: we drive our app by talking to repositories and the domain model shows the point we reached at the end of [chapter_02_repository], which covered the Repository pattern.

    +

    + +Back to our allocations project! Before: we drive our app by talking to repositories and the domain model shows the point we reached at the end of [chapter_02_repository], which covered the Repository pattern.

    @@ -171,9 +275,11 @@

    Our First Use Case: -

    Connecting Our Application to the Real World

    +

    Connecting Our Application to the Real World

    -

    Like any good agile team, we’re hustling to try to get an MVP out and +

    + +Like any good agile team, we’re hustling to try to get an MVP out and in front of the users to start gathering feedback. We have the core of our domain model and the domain service we need to allocate orders, and we have the repository interface for permanent storage.

    @@ -189,7 +295,8 @@

    Connecting Our Applicatio

    Use Flask to put an API endpoint in front of our allocate domain service. Wire up the database session and our repository. Test it with an end-to-end test and some quick-and-dirty SQL to prepare test -data.

    +data. +

  • Refactor out a service layer that can serve as an abstraction to @@ -206,9 +313,12 @@

    Connecting Our Applicatio

  • -

    A First End-to-End Test

    +

    A First End-to-End Test

    -

    No one is interested in getting into a long terminology debate about what +

    + + +No one is interested in getting into a long terminology debate about what counts as an end-to-end (E2E) test versus a functional test versus an acceptance test versus an integration test versus a unit test. Different projects need different combinations of tests, and we’ve seen perfectly successful projects just split @@ -227,22 +337,26 @@

    A First End-to-End Test

    -
    @pytest.mark.usefixtures('restart_api')
    +
    @pytest.mark.usefixtures("restart_api")
     def test_api_returns_allocation(add_stock):
    -    sku, othersku = random_sku(), random_sku('other')  #(1)
    +    sku, othersku = random_sku(), random_sku("other")  #(1)
         earlybatch = random_batchref(1)
         laterbatch = random_batchref(2)
         otherbatch = random_batchref(3)
    -    add_stock([  #(2)
    -        (laterbatch, sku, 100, '2011-01-02'),
    -        (earlybatch, sku, 100, '2011-01-01'),
    -        (otherbatch, othersku, 100, None),
    -    ])
    -    data = {'orderid': random_orderid(), 'sku': sku, 'qty': 3}
    +    add_stock(  #(2)
    +        [
    +            (laterbatch, sku, 100, "2011-01-02"),
    +            (earlybatch, sku, 100, "2011-01-01"),
    +            (otherbatch, othersku, 100, None),
    +        ]
    +    )
    +    data = {"orderid": random_orderid(), "sku": sku, "qty": 3}
         url = config.get_api_url()  #(3)
    -    r = requests.post(f'{url}/allocate', json=data)
    +
    +    r = requests.post(f"{url}/allocate", json=data)
    +
         assert r.status_code == 201
    -    assert r.json()['batchref'] == earlybatch
    + assert r.json()["batchref"] == earlybatch
    @@ -266,23 +380,26 @@

    A First End-to-End Test

    -

    Everyone solves these problems in different ways, but you’re going to need some +

    +Everyone solves these problems in different ways, but you’re going to need some way of spinning up Flask, possibly in a container, and of talking to a Postgres database. If you want to see how we did it, check out [appendix_project_structure].

    -

    The Straightforward Implementation

    +

    The Straightforward Implementation

    -

    Implementing things in the most obvious way, you might get something like this:

    +

    + +Implementing things in the most obvious way, you might get something like this:

    First cut of Flask app (flask_app.py)
    -
    from flask import Flask, jsonify, request
    +
    from flask import Flask, request
     from sqlalchemy import create_engine
     from sqlalchemy.orm import sessionmaker
     
    @@ -296,19 +413,18 @@ 

    The Straightforward Implementation< get_session = sessionmaker(bind=create_engine(config.get_postgres_uri())) app = Flask(__name__) -@app.route("/allocate", methods=['POST']) + +@app.route("/allocate", methods=["POST"]) def allocate_endpoint(): session = get_session() batches = repository.SqlAlchemyRepository(session).list() line = model.OrderLine( - request.json['orderid'], - request.json['sku'], - request.json['qty'], + request.json["orderid"], request.json["sku"], request.json["qty"], ) batchref = model.allocate(line, batches) - return jsonify({'batchref': batchref}), 201

    + return {"batchref": batchref}, 201
    @@ -318,7 +434,8 @@

    The Straightforward Implementation< nonsense, Bob and Harry, you may be thinking.

    -

    But hang on a minute—​there’s no commit. We’re not actually saving our +

    +But hang on a minute—​there’s no commit. We’re not actually saving our allocation to the database. Now we need a second test, either one that will inspect the database state after (not very black-boxy), or maybe one that checks that we can’t allocate a second line if a first should have already @@ -329,40 +446,43 @@

    The Straightforward Implementation<
    -
    @pytest.mark.usefixtures('restart_api')
    +
    @pytest.mark.usefixtures("restart_api")
     def test_allocations_are_persisted(add_stock):
         sku = random_sku()
         batch1, batch2 = random_batchref(1), random_batchref(2)
         order1, order2 = random_orderid(1), random_orderid(2)
    -    add_stock([
    -        (batch1, sku, 10, '2011-01-01'),
    -        (batch2, sku, 10, '2011-01-02'),
    -    ])
    -    line1 = {'orderid': order1, 'sku': sku, 'qty': 10}
    -    line2 = {'orderid': order2, 'sku': sku, 'qty': 10}
    +    add_stock(
    +        [(batch1, sku, 10, "2011-01-01"), (batch2, sku, 10, "2011-01-02"),]
    +    )
    +    line1 = {"orderid": order1, "sku": sku, "qty": 10}
    +    line2 = {"orderid": order2, "sku": sku, "qty": 10}
         url = config.get_api_url()
     
         # first order uses up all stock in batch 1
    -    r = requests.post(f'{url}/allocate', json=line1)
    +    r = requests.post(f"{url}/allocate", json=line1)
         assert r.status_code == 201
    -    assert r.json()['batchref'] == batch1
    +    assert r.json()["batchref"] == batch1
     
         # second order should go to batch 2
    -    r = requests.post(f'{url}/allocate', json=line2)
    +    r = requests.post(f"{url}/allocate", json=line2)
         assert r.status_code == 201
    -    assert r.json()['batchref'] == batch2
    + assert r.json()["batchref"] == batch2

    -

    Not quite so lovely, but that will force us to add the commit.

    +

    + +Not quite so lovely, but that will force us to add the commit.

    -

    Error Conditions That Require Database Checks

    +

    Error Conditions That Require Database Checks

    -

    If we keep going like this, though, things are going to get uglier and uglier.

    +

    + +If we keep going like this, though, things are going to get uglier and uglier.

    Suppose we want to add a bit of error handling. What if the domain raises an @@ -379,27 +499,27 @@

    Error Conditions That Re
    -
    @pytest.mark.usefixtures('restart_api')
    +
    @pytest.mark.usefixtures("restart_api")
     def test_400_message_for_out_of_stock(add_stock):  #(1)
    -    sku, smalL_batch, large_order = random_sku(), random_batchref(), random_orderid()
    -    add_stock([
    -        (smalL_batch, sku, 10, '2011-01-01'),
    -    ])
    -    data = {'orderid': large_order, 'sku': sku, 'qty': 20}
    +    sku, small_batch, large_order = random_sku(), random_batchref(), random_orderid()
    +    add_stock(
    +        [(small_batch, sku, 10, "2011-01-01"),]
    +    )
    +    data = {"orderid": large_order, "sku": sku, "qty": 20}
         url = config.get_api_url()
    -    r = requests.post(f'{url}/allocate', json=data)
    +    r = requests.post(f"{url}/allocate", json=data)
         assert r.status_code == 400
    -    assert r.json()['message'] == f'Out of stock for sku {sku}'
    +    assert r.json()["message"] == f"Out of stock for sku {sku}"
     
     
    -@pytest.mark.usefixtures('restart_api')
    +@pytest.mark.usefixtures("restart_api")
     def test_400_message_for_invalid_sku():  #(2)
         unknown_sku, orderid = random_sku(), random_orderid()
    -    data = {'orderid': orderid, 'sku': unknown_sku, 'qty': 20}
    +    data = {"orderid": orderid, "sku": unknown_sku, "qty": 20}
         url = config.get_api_url()
    -    r = requests.post(f'{url}/allocate', json=data)
    +    r = requests.post(f"{url}/allocate", json=data)
         assert r.status_code == 400
    -    assert r.json()['message'] == f'Invalid sku {unknown_sku}'
    + assert r.json()["message"] == f"Invalid sku {unknown_sku}"
    @@ -426,26 +546,25 @@

    Error Conditions That Re
    def is_valid_sku(sku, batches):
         return sku in {b.sku for b in batches}
     
    -@app.route("/allocate", methods=['POST'])
    +
    +@app.route("/allocate", methods=["POST"])
     def allocate_endpoint():
         session = get_session()
         batches = repository.SqlAlchemyRepository(session).list()
         line = model.OrderLine(
    -        request.json['orderid'],
    -        request.json['sku'],
    -        request.json['qty'],
    +        request.json["orderid"], request.json["sku"], request.json["qty"],
         )
     
         if not is_valid_sku(line.sku, batches):
    -        return jsonify({'message': f'Invalid sku {line.sku}'}), 400
    +        return {"message": f"Invalid sku {line.sku}"}, 400
     
         try:
             batchref = model.allocate(line, batches)
         except model.OutOfStock as e:
    -        return jsonify({'message': str(e)}), 400
    +        return {"message": str(e)}, 400
     
         session.commit()
    -    return jsonify({'batchref': batchref}), 201
    + return {"batchref": batchref}, 201

    @@ -457,9 +576,12 @@

    Error Conditions That Re

    -

    Introducing a Service Layer, and Using FakeRepository to Unit Test It

    +

    Introducing a Service Layer, and Using FakeRepository to Unit Test It

    -

    If we look at what our Flask app is doing, there’s quite a lot of what we +

    + + +If we look at what our Flask app is doing, there’s quite a lot of what we might call orchestration—fetching stuff out of our repository, validating our input against database state, handling errors, and committing in the happy path. Most of these things don’t have anything to do with having a @@ -468,11 +590,14 @@

    I end-to-end tests.

    -

    It often makes sense to split out a service layer, sometimes called an +

    + +It often makes sense to split out a service layer, sometimes called an orchestration layer or a use-case layer.

    -

    Do you remember the FakeRepository that we prepared in [chapter_03_abstractions]?

    +

    +Do you remember the FakeRepository that we prepared in [chapter_03_abstractions]?

    Our fake repository, an in-memory collection of batches (test_services.py)
    @@ -480,7 +605,6 @@

    I
    class FakeRepository(repository.AbstractRepository):
    -
         def __init__(self, batches):
             self._batches = set(batches)
     
    @@ -497,7 +621,10 @@ 

    I

    -

    Here’s where it will come in useful; it lets us test our service layer with +

    + + +Here’s where it will come in useful; it lets us test our service layer with nice, fast unit tests:

    @@ -507,7 +634,7 @@

    I
    def test_returns_allocation():
         line = model.OrderLine("o1", "COMPLICATED-LAMP", 10)
    -    batch = model.Batch("b1", "COMPLICATED-LAMP", 100, eta=None)
    +    batch = model.Batch("b1", "COMPLICATED-LAMP", 100, eta=None)
         repo = FakeRepository([batch])  #(1)
     
         result = services.allocate(line, repo, FakeSession())  #(2) (3)
    @@ -516,7 +643,7 @@ 

    I def test_error_for_invalid_sku(): line = model.OrderLine("o1", "NONEXISTENTSKU", 10) - batch = model.Batch("b1", "AREALSKU", 100, eta=None) + batch = model.Batch("b1", "AREALSKU", 100, eta=None) repo = FakeRepository([batch]) #(1) with pytest.raises(services.InvalidSku, match="Invalid sku NONEXISTENTSKU"): @@ -537,7 +664,10 @@

    I our domain model.[1]

  • -

    We also need a FakeSession to fake out the database session, as shown in the following code snippet.

    +

    We also need a FakeSession to fake out the database session, as shown in +the following code snippet. + +

  • @@ -546,11 +676,11 @@

    I
    -
    class FakeSession():
    -    committed = False
    +
    class FakeSession:
    +    committed = False
     
         def commit(self):
    -        self.committed = True
    + self.committed = True
    @@ -566,21 +696,25 @@

    I
    def test_commits():
    -    line = model.OrderLine('o1', 'OMINOUS-MIRROR', 10)
    -    batch = model.Batch('b1', 'OMINOUS-MIRROR', 100, eta=None)
    +    line = model.OrderLine("o1", "OMINOUS-MIRROR", 10)
    +    batch = model.Batch("b1", "OMINOUS-MIRROR", 100, eta=None)
         repo = FakeRepository([batch])
         session = FakeSession()
     
         services.allocate(line, repo, session)
    -    assert session.committed is True
    + assert session.committed is True

    -

    A Typical Service Function

    +

    A Typical Service Function

    -

    We’ll write a service function that looks something like this:

    +

    + + + +We’ll write a service function that looks something like this:

    Basic allocation service (services.py)
    @@ -594,10 +728,11 @@

    A Typical Service Function

    def is_valid_sku(sku, batches): return sku in {b.sku for b in batches} + def allocate(line: OrderLine, repo: AbstractRepository, session) -> str: batches = repo.list() #(1) if not is_valid_sku(line.sku, batches): #(2) - raise InvalidSku(f'Invalid sku {line.sku}') + raise InvalidSku(f"Invalid sku {line.sku}") batchref = model.allocate(line, batches) #(3) session.commit() #(4) return batchref @@ -642,17 +777,22 @@

    A Typical Service Function

    -

    It depends on a repository. We’ve chosen to make the dependency explicit, +

    + +It depends on a repository. We’ve chosen to make the dependency explicit, and we’ve used the type hint to say that we depend on AbstractRepository. This means it’ll work both when the tests give it a FakeRepository and when the Flask app gives it a SqlAlchemyRepository.

    -

    If you remember [dip], +

    +If you remember [dip], this is what we mean when we say we should "depend on abstractions." Our high-level module, the service layer, depends on the repository abstraction. And the details of the implementation for our specific choice of persistent -storage also depend on that same abstraction. See Figures #service_layer_diagram_abstract_dependencies and #service_layer_diagram_test_dependencies.

    +storage also depend on that same abstraction. See +Abstract dependencies of the service layer and +Tests provide an implementation of the abstract dependency.

    See also in [appendix_csvs] a worked example of swapping out the @@ -662,7 +802,9 @@

    A Typical Service Function

    -

    But the essentials of the service layer are there, and our Flask +

    + +But the essentials of the service layer are there, and our Flask app now looks a lot cleaner:

    @@ -670,21 +812,20 @@

    A Typical Service Function

    -
    @app.route("/allocate", methods=['POST'])
    +
    @app.route("/allocate", methods=["POST"])
     def allocate_endpoint():
         session = get_session()  #(1)
         repo = repository.SqlAlchemyRepository(session)  #(1)
         line = model.OrderLine(
    -        request.json['orderid'],  #(2)
    -        request.json['sku'],  #(2)
    -        request.json['qty'],  #(2)
    +        request.json["orderid"], request.json["sku"], request.json["qty"],  #(2)
         )
    +
         try:
             batchref = services.allocate(line, repo, session)  #(2)
         except (model.OutOfStock, services.InvalidSku) as e:
    -        return jsonify({'message': str(e)}), 400  (3)
    +        return {"message": str(e)}, 400  #(3)
     
    -    return jsonify({'batchref': batchref}), 201  (3)
    + return {"batchref": batchref}, 201 #(3)
    @@ -696,7 +837,7 @@

    A Typical Service Function

  • We extract the user’s commands from the web request and pass them -to a domain service.

    +to the service layer.

  • We return some JSON responses with the appropriate status codes.

    @@ -710,7 +851,9 @@

    A Typical Service Function

    and the domain logic stays in the domain.

  • -

    Finally, we can confidently strip down our E2E tests to just two, one for +

    + +Finally, we can confidently strip down our E2E tests to just two, one for the happy path and one for the unhappy path:

    @@ -718,32 +861,36 @@

    A Typical Service Function

    -
    @pytest.mark.usefixtures('restart_api')
    +
    @pytest.mark.usefixtures("restart_api")
     def test_happy_path_returns_201_and_allocated_batch(add_stock):
    -    sku, othersku = random_sku(), random_sku('other')
    +    sku, othersku = random_sku(), random_sku("other")
         earlybatch = random_batchref(1)
         laterbatch = random_batchref(2)
         otherbatch = random_batchref(3)
    -    add_stock([
    -        (laterbatch, sku, 100, '2011-01-02'),
    -        (earlybatch, sku, 100, '2011-01-01'),
    -        (otherbatch, othersku, 100, None),
    -    ])
    -    data = {'orderid': random_orderid(), 'sku': sku, 'qty': 3}
    +    add_stock(
    +        [
    +            (laterbatch, sku, 100, "2011-01-02"),
    +            (earlybatch, sku, 100, "2011-01-01"),
    +            (otherbatch, othersku, 100, None),
    +        ]
    +    )
    +    data = {"orderid": random_orderid(), "sku": sku, "qty": 3}
         url = config.get_api_url()
    -    r = requests.post(f'{url}/allocate', json=data)
    +
    +    r = requests.post(f"{url}/allocate", json=data)
    +
         assert r.status_code == 201
    -    assert r.json()['batchref'] == earlybatch
    +    assert r.json()["batchref"] == earlybatch
     
     
    -@pytest.mark.usefixtures('restart_api')
    +@pytest.mark.usefixtures("restart_api")
     def test_unhappy_path_returns_400_and_error_message():
         unknown_sku, orderid = random_sku(), random_orderid()
    -    data = {'orderid': orderid, 'sku': unknown_sku, 'qty': 20}
    +    data = {"orderid": orderid, "sku": unknown_sku, "qty": 20}
         url = config.get_api_url()
    -    r = requests.post(f'{url}/allocate', json=data)
    +    r = requests.post(f"{url}/allocate", json=data)
         assert r.status_code == 400
    -    assert r.json()['message'] == f'Invalid sku {unknown_sku}'
    + assert r.json()["message"] == f"Invalid sku {unknown_sku}"
    @@ -757,7 +904,8 @@

    A Typical Service Function

    Exercise for the Reader
    -

    Now that we have an allocate service, why not build out a service for +

    +Now that we have an allocate service, why not build out a service for deallocate? We’ve added an E2E test and a few stub service-layer tests for you to get started on GitHub.

    @@ -786,17 +934,23 @@

    A Typical Service Function

    -

    Why Is Everything Called a Service?

    +

    Why Is Everything Called a Service?

    -

    Some of you are probably scratching your heads at this point trying to figure +

    + + + +Some of you are probably scratching your heads at this point trying to figure out exactly what the difference is between a domain service and a service layer.

    -

    We’re sorry—we didn’t choose the names, or we’d have much cooler and friendlier +

    +We’re sorry—we didn’t choose the names, or we’d have much cooler and friendlier ways to talk about this stuff.

    -

    We’re using two things called a service in this chapter. The first is an +

    +We’re using two things called a service in this chapter. The first is an application service (our service layer). Its job is to handle requests from the outside world and to orchestrate an operation. What we mean is that the service layer drives the application by following a bunch of simple steps:

    @@ -819,7 +973,8 @@

    Why Is Everything Called a Service?

    system, and keeping it separate from business logic helps to keep things tidy.

    -

    The second type of service is a domain service. This is the name for a piece of +

    +The second type of service is a domain service. This is the name for a piece of logic that belongs in the domain model but doesn’t sit naturally inside a stateful entity or value object. For example, if you were building a shopping cart application, you might choose to build taxation rules as a domain service. @@ -830,9 +985,13 @@

    Why Is Everything Called a Service?

    -

    Putting Things in Folders to See Where It All Belongs

    +

    Putting Things in Folders to See Where It All Belongs

    -

    As our application gets bigger, we’ll need to keep tidying our directory +

    + + + +As our application gets bigger, we’ll need to keep tidying our directory structure. The layout of our project gives us useful hints about what kinds of object we’ll find in each file.

    @@ -882,36 +1041,46 @@

    Putting Things i but for a more complex application, you might have one file per class; you might have helper parent classes for Entity, ValueObject, and Aggregate, and you might add an exceptions.py for domain-layer exceptions -and, as you’ll see in [part2], commands.py and events.py.

    +and, as you’ll see in [part2], commands.py and events.py. +

  • We’ll distinguish the service layer. Currently that’s just one file called services.py for our service-layer functions. You could -add service-layer exceptions here, and as you’ll see in [chapter_05_high_gear_low_gear], we’ll add unit_of_work.py.

    +add service-layer exceptions here, and as you’ll see in +[chapter_05_high_gear_low_gear], we’ll add unit_of_work.py.

  • Adapters is a nod to the ports and adapters terminology. This will fill up with any other abstractions around external I/O (e.g., a redis_client.py). Strictly speaking, you would call these secondary adapters or driven -adapters, or sometimes inward-facing adapters.

    +adapters, or sometimes inward-facing adapters. + + + +

  • Entrypoints are the places we drive our application from. In the official ports and adapters terminology, these are adapters too, and are -referred to as primary, driving, or outward-facing adapters.

    +referred to as primary, driving, or outward-facing adapters. +

  • -

    What about ports? As you may remember, they are the abstract interfaces that the +

    +What about ports? As you may remember, they are the abstract interfaces that the adapters implement. We tend to keep them in the same file as the adapters that implement them.

    -

    Wrap-Up

    +

    Wrap-Up

    -

    Adding the service layer has really bought us quite a lot:

    +

    + +Adding the service layer has really bought us quite a lot:

      @@ -940,19 +1109,26 @@

      Wrap-Up

    -

    The DIP in Action

    +

    The DIP in Action

    -

    Abstract dependencies of the service layer shows the +

    + + +Abstract dependencies of the service layer shows the dependencies of our service layer: the domain model and AbstractRepository (the port, in ports and adapters terminology).

    -

    When we run the tests, Tests provide an implementation of the abstract dependency shows +

    + +When we run the tests, Tests provide an implementation of the abstract dependency shows how we implement the abstract dependencies by using FakeRepository (the adapter).

    -

    And when we actually run our app, we swap in the "real" dependency shown in +

    + +And when we actually run our app, we swap in the "real" dependency shown in Dependencies at runtime.

    @@ -1052,7 +1228,9 @@

    The DIP in Action

    Wonderful.

    -

    Let’s pause for Service layer: the trade-offs, +

    + +Let’s pause for Service layer: the trade-offs, in which we consider the pros and cons of having a service layer at all.

    @@ -1101,14 +1279,18 @@

    The DIP in Action

  • Putting too much logic into the service layer can lead to the Anemic Domain -anti-pattern. It’s better to introduce this layer after you spot orchestration -logic creeping into your controllers.

    +antipattern. It’s better to introduce this layer after you spot orchestration +logic creeping into your controllers. + +

  • You can get a lot of the benefits that come from having rich domain models by simply pushing logic out of your controllers and down to the model layer, without needing to add an extra layer in between (aka "fat models, thin -controllers").

    +controllers"). + +

  • @@ -1138,6 +1320,11 @@

    The DIP in Action

    +

    @@ -1147,93 +1334,22 @@

    The DIP in Action

    -
    diff --git a/docs/book/chapter_05_high_gear_low_gear.html b/book/chapter_05_high_gear_low_gear.html similarity index 80% rename from docs/book/chapter_05_high_gear_low_gear.html rename to book/chapter_05_high_gear_low_gear.html index 0c1aee4..5de54f1 100644 --- a/docs/book/chapter_05_high_gear_low_gear.html +++ b/book/chapter_05_high_gear_low_gear.html @@ -3,7 +3,7 @@ - + TDD in High Gear and Low Gear + @@ -81,7 +183,7 @@
    -

    TDD in High Gear and Low Gear

    +

    5: TDD in High Gear and Low Gear

    -

    We’ve introduced the service layer to capture some of the additional +

    +We’ve introduced the service layer to capture some of the additional orchestration responsibilities we need from a working application. The service layer helps us clearly define our use cases and the workflow for each: what we need to get from our repositories, what pre-checks and current state validation we should do, and what we save at the end.

    -

    But currently, many of our unit tests operate at a lower level, acting +

    +But currently, many of our unit tests operate at a lower level, acting directly on the model. In this chapter we’ll discuss the trade-offs involved in moving those tests up to the service-layer level, and some more general testing guidelines.

    @@ -126,7 +230,8 @@

    TDD in High Gear and Low Gear

    Harry Says: Seeing a Test Pyramid in Action Was a Light-Bulb Moment
    -

    Here are a few words from Harry directly:

    +

    +Here are a few words from Harry directly:

    I was initially skeptical of all Bob’s architectural patterns, but seeing @@ -147,9 +252,11 @@

    TDD in High Gear and Low Gear

    -

    How Is Our Test Pyramid Looking?

    +

    How Is Our Test Pyramid Looking?

    -

    Let’s see what this move to using a service layer, with its own service-layer tests, +

    + +Let’s see what this move to using a service layer, with its own service-layer tests, does to our test pyramid:

    @@ -157,7 +264,7 @@

    How Is Our Test Pyramid Looking?

    -

    Should Domain Layer Tests Move to the Service Layer?

    +

    Should Domain Layer Tests Move to the Service Layer?

    -

    Let’s see what happens if we take this a step further. Since we can test our +

    + + +Let’s see what happens if we take this a step further. Since we can test our software against the service layer, we don’t really need tests for the domain model anymore. Instead, we could rewrite all of the domain-level tests from [chapter_01_domain_model] in terms of the service layer:

    @@ -190,7 +300,7 @@

    Should Domain Laye
    # domain-layer test:
     def test_prefers_current_stock_batches_to_shipments():
    -    in_stock_batch = Batch("in-stock-batch", "RETRO-CLOCK", 100, eta=None)
    +    in_stock_batch = Batch("in-stock-batch", "RETRO-CLOCK", 100, eta=None)
         shipment_batch = Batch("shipment-batch", "RETRO-CLOCK", 100, eta=tomorrow)
         line = OrderLine("oref", "RETRO-CLOCK", 10)
     
    @@ -202,7 +312,7 @@ 

    Should Domain Laye # service-layer test: def test_prefers_warehouse_batches_to_shipments(): - in_stock_batch = Batch("in-stock-batch", "RETRO-CLOCK", 100, eta=None) + in_stock_batch = Batch("in-stock-batch", "RETRO-CLOCK", 100, eta=None) shipment_batch = Batch("shipment-batch", "RETRO-CLOCK", 100, eta=tomorrow) repo = FakeRepository([in_stock_batch, shipment_batch]) session = FakeSession() @@ -218,7 +328,9 @@

    Should Domain Laye

    -

    Why would we want to do that?

    +

    + +Why would we want to do that?

    Tests are supposed to help us change our system fearlessly, but often @@ -261,9 +373,12 @@

    Should Domain Laye

    -

    On Deciding What Kind of Tests to Write

    +

    On Deciding What Kind of Tests to Write

    -

    You might be asking yourself, "Should I rewrite all my unit tests, then? Is it +

    + + +You might be asking yourself, "Should I rewrite all my unit tests, then? Is it wrong to write tests against the domain model?" To answer those questions, it’s important to understand the trade-off between coupling and design feedback (see The test spectrum).

    @@ -287,7 +402,8 @@

    On Deciding What Kind of Tests to Write

    -

    Extreme programming (XP) exhorts us to "listen to the code." When we’re writing +

    +Extreme programming (XP) exhorts us to "listen to the code." When we’re writing tests, we might find that the code is hard to use or notice a code smell. This is a trigger for us to refactor, and to reconsider our design.

    @@ -322,19 +438,22 @@

    On Deciding What Kind of Tests to Write

    -

    High and Low Gear

    +

    High and Low Gear

    -

    Most of the time, when we are adding a new feature or fixing a bug, we don’t +

    +Most of the time, when we are adding a new feature or fixing a bug, we don’t need to make extensive changes to the domain model. In these cases, we prefer to write tests against services because of the lower coupling and higher coverage.

    -

    For example, when writing an add_stock function or a cancel_order feature, +

    +For example, when writing an add_stock function or a cancel_order feature, we can work more quickly and with less coupling by writing tests against the service layer.

    -

    When starting a new project or when hitting a particularly gnarly problem, +

    +When starting a new project or when hitting a particularly gnarly problem, we will drop back down to writing tests against the domain model so we get better feedback and executable documentation of our intent.

    @@ -347,9 +466,12 @@

    High and Low Gear

    -

    Fully Decoupling the Service-Layer Tests from the Domain

    +

    Fully Decoupling the Service-Layer Tests from the Domain

    -

    We still have direct dependencies on the domain in our service-layer +

    + + +We still have direct dependencies on the domain in our service-layer tests, because we use domain objects to set up our test data and to invoke our service-layer functions.

    @@ -379,7 +501,8 @@

    Fully Decoupling the Service-Layer Tests from the D
    def allocate(
    -        orderid: str, sku: str, qty: int, repo: AbstractRepository, session
    +    orderid: str, sku: str, qty: int,
    +    repo: AbstractRepository, session
     ) -> str:
    @@ -394,7 +517,7 @@

    Fully Decoupling the Service-Layer Tests from the D
    def test_returns_allocation():
    -    batch = model.Batch("batch1", "COMPLICATED-LAMP", 100, eta=None)
    +    batch = model.Batch("batch1", "COMPLICATED-LAMP", 100, eta=None)
         repo = FakeRepository([batch])
     
         result = services.allocate("o1", "COMPLICATED-LAMP", 10, repo, FakeSession())
    @@ -409,9 +532,13 @@ 

    Fully Decoupling the Service-Layer Tests from the D model works, we’ll have to change a bunch of tests.

    -

    Mitigation: Keep All Domain Dependencies in Fixture Functions

    +

    Mitigation: Keep All Domain Dependencies in Fixture Functions

    -

    We could at least abstract that out to a helper function or a fixture +

    + + + +We could at least abstract that out to a helper function or a fixture in our tests. Here’s one way you could do that, adding a factory function on FakeRepository:

    @@ -423,7 +550,7 @@

    Mitigatio
    class FakeRepository(set):
     
         @staticmethod
    -    def for_batch(ref, sku, qty, eta=None):
    +    def for_batch(ref, sku, qty, eta=None):
             return FakeRepository([
                 model.Batch(ref, sku, qty, eta),
             ])
    @@ -432,7 +559,7 @@ 

    Mitigatio def test_returns_allocation(): - repo = FakeRepository.for_batch("batch1", "COMPLICATED-LAMP", 100, eta=None) + repo = FakeRepository.for_batch("batch1", "COMPLICATED-LAMP", 100, eta=None) result = services.allocate("o1", "COMPLICATED-LAMP", 10, repo, FakeSession()) assert result == "batch1"

    @@ -445,9 +572,10 @@

    Mitigatio

    -

    Adding a Missing Service

    +

    Adding a Missing Service

    -

    We could go one step further, though. If we had a service to add stock, +

    +We could go one step further, though. If we had a service to add stock, we could use that and make our service-layer tests fully expressed in terms of the service layer’s official use cases, removing all dependencies on the domain:

    @@ -459,8 +587,8 @@

    Adding a Missing Service

    def test_add_batch():
         repo, session = FakeRepository([]), FakeSession()
    -    services.add_batch("b1", "CRUNCHY-ARMCHAIR", 100, None, repo, session)
    -    assert repo.get("b1") is not None
    +    services.add_batch("b1", "CRUNCHY-ARMCHAIR", 100, None, repo, session)
    +    assert repo.get("b1") is not None
         assert session.committed
    @@ -489,17 +617,17 @@

    Adding a Missing Service

    def add_batch(
    -        ref: str, sku: str, qty: int, eta: Optional[date],
    -        repo: AbstractRepository, session,
    -):
    +    ref: str, sku: str, qty: int, eta: Optional[date],
    +    repo: AbstractRepository, session,
    +) -> None:
         repo.add(model.Batch(ref, sku, qty, eta))
         session.commit()
     
     
     def allocate(
    -        orderid: str, sku: str, qty: int, repo: AbstractRepository, session
    -) -> str:
    -    ...
    + orderid: str, sku: str, qty: int, + repo: AbstractRepository, session +) -> str:
    @@ -519,7 +647,8 @@

    Adding a Missing Service

    -

    That now allows us to rewrite all of our service-layer tests purely +

    +That now allows us to rewrite all of our service-layer tests purely in terms of the services themselves, using only primitives, and without any dependencies on the model:

    @@ -530,14 +659,14 @@

    Adding a Missing Service

    def test_allocate_returns_allocation():
         repo, session = FakeRepository([]), FakeSession()
    -    services.add_batch("batch1", "COMPLICATED-LAMP", 100, None, repo, session)
    +    services.add_batch("batch1", "COMPLICATED-LAMP", 100, None, repo, session)
         result = services.allocate("o1", "COMPLICATED-LAMP", 10, repo, session)
         assert result == "batch1"
     
     
     def test_allocate_errors_for_invalid_sku():
         repo, session = FakeRepository([]), FakeSession()
    -    services.add_batch("b1", "AREALSKU", 100, None, repo, session)
    +    services.add_batch("b1", "AREALSKU", 100, None, repo, session)
     
         with pytest.raises(services.InvalidSku, match="Invalid sku NONEXISTENTSKU"):
             services.allocate("o1", "NONEXISTENTSKU", 10, repo, FakeSession())
    @@ -546,16 +675,23 @@

    Adding a Missing Service

    -

    This is a really nice place to be in. Our service-layer tests depend on only +

    + + +This is a really nice place to be in. Our service-layer tests depend on only the service layer itself, leaving us completely free to refactor the model as we see fit.

    -

    Carrying the Improvement Through to the E2E Tests

    +

    Carrying the Improvement Through to the E2E Tests

    -

    In the same way that adding add_batch helped decouple our service-layer +

    + + + +In the same way that adding add_batch helped decouple our service-layer tests from the model, adding an API endpoint to add a batch would remove the need for the ugly add_stock fixture, and our E2E tests could be free of those hardcoded SQL queries and the direct dependency on the database.

    @@ -569,18 +705,22 @@

    Carrying the Improve
    -
    @app.route("/add_batch", methods=['POST'])
    +
    @app.route("/add_batch", methods=["POST"])
     def add_batch():
         session = get_session()
         repo = repository.SqlAlchemyRepository(session)
    -    eta = request.json['eta']
    -    if eta is not None:
    +    eta = request.json["eta"]
    +    if eta is not None:
             eta = datetime.fromisoformat(eta).date()
         services.add_batch(
    -        request.json['ref'], request.json['sku'], request.json['qty'], eta,
    -        repo, session
    +        request.json["ref"],
    +        request.json["sku"],
    +        request.json["qty"],
    +        eta,
    +        repo,
    +        session,
         )
    -    return 'OK', 201
    + return "OK", 201
    @@ -614,36 +754,39 @@

    Carrying the Improve
    def post_to_add_batch(ref, sku, qty, eta):
         url = config.get_api_url()
         r = requests.post(
    -        f'{url}/add_batch',
    -        json={'ref': ref, 'sku': sku, 'qty': qty, 'eta': eta}
    +        f"{url}/add_batch", json={"ref": ref, "sku": sku, "qty": qty, "eta": eta}
         )
         assert r.status_code == 201
     
     
    -@pytest.mark.usefixtures('postgres_db')
    -@pytest.mark.usefixtures('restart_api')
    +@pytest.mark.usefixtures("postgres_db")
    +@pytest.mark.usefixtures("restart_api")
     def test_happy_path_returns_201_and_allocated_batch():
    -    sku, othersku = random_sku(), random_sku('other')
    +    sku, othersku = random_sku(), random_sku("other")
         earlybatch = random_batchref(1)
         laterbatch = random_batchref(2)
         otherbatch = random_batchref(3)
    -    post_to_add_batch(laterbatch, sku, 100, '2011-01-02')
    -    post_to_add_batch(earlybatch, sku, 100, '2011-01-01')
    -    post_to_add_batch(otherbatch, othersku, 100, None)
    -    data = {'orderid': random_orderid(), 'sku': sku, 'qty': 3}
    +    post_to_add_batch(laterbatch, sku, 100, "2011-01-02")
    +    post_to_add_batch(earlybatch, sku, 100, "2011-01-01")
    +    post_to_add_batch(otherbatch, othersku, 100, None)
    +    data = {"orderid": random_orderid(), "sku": sku, "qty": 3}
    +
         url = config.get_api_url()
    -    r = requests.post(f'{url}/allocate', json=data)
    +    r = requests.post(f"{url}/allocate", json=data)
    +
         assert r.status_code == 201
    -    assert r.json()['batchref'] == earlybatch
    + assert r.json()["batchref"] == earlybatch

    -

    Wrap-Up

    +

    Wrap-Up

    -

    Once you have a service layer in place, you really can move the majority +

    + +Once you have a service layer in place, you really can move the majority of your test coverage to unit tests and develop a healthy test pyramid.

    @@ -655,21 +798,23 @@

    Wrap-Up

    This might be written against an HTTP API, for example. The objective is to demonstrate that the feature works, and that all the moving parts -are glued together correctly.

    +are glued together correctly. +

    Write the bulk of your tests against the service layer
    -

    These edge-to-edge tests offer a good trade-off between coverage, - runtime, and efficiency. Each test tends to cover one code path of a - feature and use fakes for I/O. This is the place to exhaustively - cover all the edge cases and the ins and outs of your business logic.[1] and -[fake_message_bus].]

    +

    These edge-to-edge tests offer a good trade-off between coverage, +runtime, and efficiency. Each test tends to cover one code path of a +feature and use fakes for I/O. This is the place to exhaustively +cover all the edge cases and the ins and outs of your business logic.[1] +

    Maintain a small core of tests written against your domain model

    These tests have highly focused coverage and are more brittle, but they have the highest feedback. Don’t be afraid to delete these tests if the -functionality is later covered by tests at the service layer.

    +functionality is later covered by tests at the service layer. +

    Error handling counts as a feature
    @@ -677,7 +822,9 @@

    Wrap-Up

    bubble up to your entrypoints (e.g., Flask) are handled in the same way. This means you need to test only the happy path for each feature, and to reserve one end-to-end test for all unhappy paths (and many unhappy path -unit tests, of course).

    +unit tests, of course). + +

    @@ -695,7 +842,8 @@

    Wrap-Up

  • In an ideal world, you’ll have all the services you need to be able to test entirely against the service layer, rather than hacking state via -repositories or the database. This pays off in your end-to-end tests as well.

    +repositories or the database. This pays off in your end-to-end tests as well. +

  • @@ -705,102 +853,36 @@

    Wrap-Up

    +

    -1. A valid concern about writing tests at a higher level is that it can lead to combinatorial explosion for more complex use cases. In these cases, dropping down to lower-level unit tests of the various collaborating domain objects can be useful. But see also [chapter_08_events_and_message_bus -
    +1. A valid concern about writing tests at a higher level is that it can lead to combinatorial explosion for more complex use cases. In these cases, dropping down to lower-level unit tests of the various collaborating domain objects can be useful. But see also [chapter_08_events_and_message_bus] and [fake_message_bus]. +
    -
    diff --git a/docs/book/chapter_06_uow.html b/book/chapter_06_uow.html similarity index 81% rename from docs/book/chapter_06_uow.html rename to book/chapter_06_uow.html index 79f91b5..f316124 100644 --- a/docs/book/chapter_06_uow.html +++ b/book/chapter_06_uow.html @@ -3,7 +3,7 @@ - + Unit of Work Pattern + @@ -81,7 +183,7 @@
    -

    Unit of Work Pattern

    +

    6: Unit of Work Pattern

    -

    In this chapter we’ll introduce the final piece of the puzzle that ties +

    +In this chapter we’ll introduce the final piece of the puzzle that ties together the Repository and Service Layer patterns: the Unit of Work pattern.

    -

    If the Repository pattern is our abstraction over the idea of persistent storage, +

    + +If the Repository pattern is our abstraction over the idea of persistent storage, the Unit of Work (UoW) pattern is our abstraction over the idea of atomic operations. It will allow us to finally and fully decouple our service layer from the data layer.

    -

    Without UoW: API talks directly to three layers shows that, currently, a lot of communication occurs +

    + +Without UoW: API talks directly to three layers shows that, currently, a lot of communication occurs across the layers of our infrastructure: the API talks directly to the database layer to start a session, it talks to the repository layer to initialize SQLAlchemyRepository, and it talks to the service layer to ask it to allocate.

    @@ -155,14 +262,17 @@

    Unit of Work Pattern

    Figure 1. Without UoW: API talks directly to three layers
    -

    With UoW: UoW now manages database state shows our target state. The Flask API now does only two +

    + +With UoW: UoW now manages database state shows our target state. The Flask API now does only two things: it initializes a unit of work, and it invokes a service. The service collaborates with the UoW (we like to think of the UoW as being part of the service layer), but neither the service function itself nor Flask now needs to talk directly to the database.

    -

    And we’ll do it all using a lovely piece of Python syntax, a context manager.

    +

    +And we’ll do it all using a lovely piece of Python syntax, a context manager.

    @@ -171,9 +281,11 @@

    Unit of Work Pattern

    Figure 2. With UoW: UoW now manages database state
    -

    The Unit of Work Collaborates with the Repository

    +

    The Unit of Work Collaborates with the Repository

    -

    Let’s see the unit of work (or UoW, which we pronounce "you-wow") in action. Here’s how the service layer will look when we’re finished:

    +

    + +Let’s see the unit of work (or UoW, which we pronounce "you-wow") in action. Here’s how the service layer will look when we’re finished:

    Preview of unit of work in action (src/allocation/service_layer/services.py)
    @@ -181,8 +293,8 @@

    The Unit of Work Col
    def allocate(
    -        orderid: str, sku: str, qty: int,
    -        uow: unit_of_work.AbstractUnitOfWork
    +    orderid: str, sku: str, qty: int,
    +    uow: unit_of_work.AbstractUnitOfWork,
     ) -> str:
         line = OrderLine(orderid, sku, qty)
         with uow:  #(1)
    @@ -197,11 +309,13 @@ 

    The Unit of Work Col
    1. -

      We’ll start a UoW as a context manager.

      +

      We’ll start a UoW as a context manager. +

    2. uow.batches is the batches repo, so the UoW provides us -access to our permanent storage.

      +access to our permanent storage. +

    3. When we’re done, we commit or roll back our work, using the UoW.

      @@ -209,7 +323,9 @@

      The Unit of Work Col

    -

    The UoW acts as a single entrypoint to our persistent storage, and it +

    + +The UoW acts as a single entrypoint to our persistent storage, and it keeps track of what objects were loaded and of the latest state.[1]

    @@ -233,9 +349,12 @@

    The Unit of Work Col

    -

    Test-Driving a UoW with Integration Tests

    +

    Test-Driving a UoW with Integration Tests

    -

    Here are our integration tests for the UOW:

    +

    + + +Here are our integration tests for the UOW:

    A basic "round-trip" test for a UoW (tests/integration/test_uow.py)
    @@ -244,18 +363,18 @@

    Test-Driving a UoW with Inte
    def test_uow_can_retrieve_a_batch_and_allocate_to_it(session_factory):
         session = session_factory()
    -    insert_batch(session, 'batch1', 'HIPSTER-WORKBENCH', 100, None)
    +    insert_batch(session, "batch1", "HIPSTER-WORKBENCH", 100, None)
         session.commit()
     
         uow = unit_of_work.SqlAlchemyUnitOfWork(session_factory)  #(1)
         with uow:
    -        batch = uow.batches.get(reference='batch1')  #(2)
    -        line = model.OrderLine('o1', 'HIPSTER-WORKBENCH', 10)
    +        batch = uow.batches.get(reference="batch1")  #(2)
    +        line = model.OrderLine("o1", "HIPSTER-WORKBENCH", 10)
             batch.allocate(line)
             uow.commit()  #(3)
     
    -    batchref = get_allocated_batch_ref(session, 'o1', 'HIPSTER-WORKBENCH')
    -    assert batchref == 'batch1'
    + batchref = get_allocated_batch_ref(session, "o1", "HIPSTER-WORKBENCH") + assert batchref == "batch1"

    @@ -276,8 +395,9 @@

    Test-Driving a UoW with Inte

    -

    For the curious, the insert_batch and get_allocated_batch_ref helpers -look like this:

    +

    +For the curious, the insert_batch and get_allocated_batch_ref helpers look +like this:

    Helpers for doing SQL stuff (tests/integration/test_uow.py)
    @@ -286,32 +406,50 @@

    Test-Driving a UoW with Inte
    def insert_batch(session, ref, sku, qty, eta):
         session.execute(
    -        'INSERT INTO batches (reference, sku, _purchased_quantity, eta)'
    -        ' VALUES (:ref, :sku, :qty, :eta)',
    -        dict(ref=ref, sku=sku, qty=qty, eta=eta)
    +        "INSERT INTO batches (reference, sku, _purchased_quantity, eta)"
    +        " VALUES (:ref, :sku, :qty, :eta)",
    +        dict(ref=ref, sku=sku, qty=qty, eta=eta),
         )
     
     
     def get_allocated_batch_ref(session, orderid, sku):
    -    [[orderlineid]] = session.execute(
    -        'SELECT id FROM order_lines WHERE orderid=:orderid AND sku=:sku',
    -        dict(orderid=orderid, sku=sku)
    +    [[orderlineid]] = session.execute(  #(1)
    +        "SELECT id FROM order_lines WHERE orderid=:orderid AND sku=:sku",
    +        dict(orderid=orderid, sku=sku),
         )
    -    [[batchref]] = session.execute(
    -        'SELECT b.reference FROM allocations JOIN batches AS b ON batch_id = b.id'
    -        ' WHERE orderline_id=:orderlineid',
    -        dict(orderlineid=orderlineid)
    +    [[batchref]] = session.execute(  #(1)
    +        "SELECT b.reference FROM allocations JOIN batches AS b ON batch_id = b.id"
    +        " WHERE orderline_id=:orderlineid",
    +        dict(orderlineid=orderlineid),
         )
         return batchref

    +
    +
      +
    1. +

      The = syntax is a little too-clever-by-half, apologies. +What’s happening is that session.execute returns a list of rows, +where each row is a tuple of column values; +in our specific case, it’s a list of one row, +which is a tuple with one column value in. +The double-square-bracket on the left hand side +is doing (double) assignment-unpacking to get the single value +back out of these two nested sequences. +It becomes readable once you’ve used it a few times!

      +
    2. +
    +
    -

    Unit of Work and Its Context Manager

    +

    Unit of Work and Its Context Manager

    -

    In our tests we’ve implicitly defined an interface for what a UoW needs to do. Let’s make that explicit by using an abstract +

    + + +In our tests we’ve implicitly defined an interface for what a UoW needs to do. Let’s make that explicit by using an abstract base class:

    @@ -325,11 +463,11 @@

    Unit of Work and Its Context Mana def __exit__(self, *args): #(2) self.rollback() #(4) - @abc.abstractmethod + @abc.abstractmethod def commit(self): #(3) raise NotImplementedError - @abc.abstractmethod + @abc.abstractmethod def rollback(self): #(4) raise NotImplementedError

    @@ -345,7 +483,9 @@

    Unit of Work and Its Context Mana
  • If you’ve never seen a context manager, __enter__ and __exit__ are the two magic methods that execute when we enter the with block and -when we exit it, respectively. They’re our setup and teardown phases.

    +when we exit it, respectively. They’re our setup and teardown phases. + +

  • We’ll call this method to explicitly commit our work when we’re ready.

    @@ -353,14 +493,18 @@

    Unit of Work and Its Context Mana
  • If we don’t commit, or if we exit the context manager by raising an error, we do a rollback. (The rollback has no effect if commit() has been -called. Read on for more discussion of this.)

    +called. Read on for more discussion of this.) +

  • -

    The Real Unit of Work Uses SQLAlchemy Sessions

    +

    The Real Unit of Work Uses SQLAlchemy Sessions

    -

    The main thing that our concrete implementation adds is the +

    + + +The main thing that our concrete implementation adds is the database session:

    @@ -368,12 +512,14 @@

    The Real Unit of Work U
    -
    DEFAULT_SESSION_FACTORY = sessionmaker(bind=create_engine(  #(1)
    -    config.get_postgres_uri(),
    -))
    +
    DEFAULT_SESSION_FACTORY = sessionmaker(  #(1)
    +    bind=create_engine(
    +        config.get_postgres_uri(),
    +    )
    +)
     
    -class SqlAlchemyUnitOfWork(AbstractUnitOfWork):
     
    +class SqlAlchemyUnitOfWork(AbstractUnitOfWork):
         def __init__(self, session_factory=DEFAULT_SESSION_FACTORY):
             self.session_factory = session_factory  #(1)
     
    @@ -404,22 +550,28 @@ 

    The Real Unit of Work U
  • The __enter__ method is responsible for starting a database session and instantiating -a real repository that can use that session.

    +a real repository that can use that session. +

  • We close the session on exit.

  • Finally, we provide concrete commit() and rollback() methods that -use our database session.

    +use our database session. + +

  • -

    Fake Unit of Work for Testing

    +

    Fake Unit of Work for Testing

    -

    Here’s how we use a fake UoW in our service-layer tests:

    +

    + + +Here’s how we use a fake UoW in our service-layer tests:

    Fake UoW (tests/unit/test_services.py)
    @@ -427,29 +579,27 @@

    Fake Unit of Work for Testing

    class FakeUnitOfWork(unit_of_work.AbstractUnitOfWork):
    -
         def __init__(self):
             self.batches = FakeRepository([])  #(1)
    -        self.committed = False  #(2)
    +        self.committed = False  #(2)
     
         def commit(self):
    -        self.committed = True  #(2)
    +        self.committed = True  #(2)
     
         def rollback(self):
             pass
     
     
    -
     def test_add_batch():
         uow = FakeUnitOfWork()  #(3)
    -    services.add_batch("b1", "CRUNCHY-ARMCHAIR", 100, None, uow)  #(3)
    -    assert uow.batches.get("b1") is not None
    +    services.add_batch("b1", "CRUNCHY-ARMCHAIR", 100, None, uow)  #(3)
    +    assert uow.batches.get("b1") is not None
         assert uow.committed
     
     
     def test_allocate_returns_allocation():
         uow = FakeUnitOfWork()  #(3)
    -    services.add_batch("batch1", "COMPLICATED-LAMP", 100, None, uow)  #(3)
    +    services.add_batch("batch1", "COMPLICATED-LAMP", 100, None, uow)  #(3)
         result = services.allocate("o1", "COMPLICATED-LAMP", 10, uow)  #(3)
         assert result == "batch1"
     ...
    @@ -482,7 +632,9 @@

    Fake Unit of Work for Testing

    Don’t Mock What You Don’t Own
    -

    Why do we feel more comfortable mocking the UoW than the session? +

    + +Why do we feel more comfortable mocking the UoW than the session? Both of our fakes achieve the same thing: they give us a way to swap out our persistence layer so we can run tests in memory instead of needing to talk to a real database. The difference is in the resulting design.

    @@ -507,16 +659,19 @@

    Fake Unit of Work for Testing

    "Don’t mock what you don’t own" is a rule of thumb that forces us to build these simple abstractions over messy subsystems. This has the same performance benefit as mocking the SQLAlchemy session but encourages us to think carefully -about our designs.

    +about our designs. +

    -

    Using the UoW in the Service Layer

    +

    Using the UoW in the Service Layer

    -

    Here’s what our new service layer looks like:

    +

    + +Here’s what our new service layer looks like:

    Service layer using UoW (src/allocation/service_layer/services.py)
    @@ -524,8 +679,8 @@

    Using the UoW in the Service Layer<
    def add_batch(
    -        ref: str, sku: str, qty: int, eta: Optional[date],
    -        uow: unit_of_work.AbstractUnitOfWork  #(1)
    +    ref: str, sku: str, qty: int, eta: Optional[date],
    +    uow: unit_of_work.AbstractUnitOfWork,  #(1)
     ):
         with uow:
             uow.batches.add(model.Batch(ref, sku, qty, eta))
    @@ -533,14 +688,14 @@ 

    Using the UoW in the Service Layer< def allocate( - orderid: str, sku: str, qty: int, - uow: unit_of_work.AbstractUnitOfWork #(1) + orderid: str, sku: str, qty: int, + uow: unit_of_work.AbstractUnitOfWork, #(1) ) -> str: line = OrderLine(orderid, sku, qty) with uow: batches = uow.batches.list() if not is_valid_sku(line.sku, batches): - raise InvalidSku(f'Invalid sku {line.sku}') + raise InvalidSku(f"Invalid sku {line.sku}") batchref = model.allocate(line, batches) uow.commit() return batchref

    @@ -551,16 +706,21 @@

    Using the UoW in the Service Layer<
    1. -

      Our service layer now has only the one dependency, once again -on an abstract UoW.

      +

      Our service layer now has only the one dependency, +once again on an abstract UoW. +

    -

    Explicit Tests for Commit/Rollback Behavior

    +

    Explicit Tests for Commit/Rollback Behavior

    -

    To convince ourselves that the commit/rollback behavior works, we wrote +

    + + + +To convince ourselves that the commit/rollback behavior works, we wrote a couple of tests:

    @@ -571,7 +731,7 @@

    Explicit Tests for Commit/R
    def test_rolls_back_uncommitted_work_by_default(session_factory):
         uow = unit_of_work.SqlAlchemyUnitOfWork(session_factory)
         with uow:
    -        insert_batch(uow.session, 'batch1', 'MEDIUM-PLINTH', 100, None)
    +        insert_batch(uow.session, "batch1", "MEDIUM-PLINTH", 100, None)
     
         new_session = session_factory()
         rows = list(new_session.execute('SELECT * FROM "batches"'))
    @@ -585,7 +745,7 @@ 

    Explicit Tests for Commit/R uow = unit_of_work.SqlAlchemyUnitOfWork(session_factory) with pytest.raises(MyException): with uow: - insert_batch(uow.session, 'batch1', 'LARGE-FORK', 100, None) + insert_batch(uow.session, "batch1", "LARGE-FORK", 100, None) raise MyException() new_session = session_factory() @@ -608,15 +768,19 @@

    Explicit Tests for Commit/R SQLite instead of Postgres, but in [chapter_07_aggregate], we’ll switch some of the tests to using the real database. It’s convenient that our UoW class makes that easy! +

    -

    Explicit Versus Implicit Commits

    +

    Explicit Versus Implicit Commits

    -

    Now we briefly digress on different ways of implementing the UoW pattern.

    +

    + + +Now we briefly digress on different ways of implementing the UoW pattern.

    We could imagine a slightly different version of the UoW that commits by default @@ -634,7 +798,7 @@

    Explicit Versus Implicit Commits

    return self def __exit__(self, exn_type, exn_value, traceback): - if exn_type is None: + if exn_type is None: self.commit() #(1) else: self.rollback() #(2)
    @@ -687,27 +851,34 @@

    Explicit Versus Implicit Commits

    -

    Examples: Using UoW to Group Multiple Operations into an Atomic Unit

    +

    Examples: Using UoW to Group Multiple Operations into an Atomic Unit

    -

    Here are a few examples showing the Unit of Work pattern in use. You can +

    + +Here are a few examples showing the Unit of Work pattern in use. You can see how it leads to simple reasoning about what blocks of code happen together.

    -

    Example 1: Reallocate

    +

    Example 1: Reallocate

    -

    Suppose we want to be able to deallocate and then reallocate orders:

    +

    + +Suppose we want to be able to deallocate and then reallocate orders:

    Reallocate service function
    -

    Example 2: Change Batch Quantity

    +

    Example 2: Change Batch Quantity

    -

    Our shipping company gives us a call to say that one of the container doors +

    +Our shipping company gives us a call to say that one of the container doors opened, and half our sofas have fallen into the Indian Ocean. Oops!

    @@ -738,7 +910,10 @@

    Example 2: Change Batch Quantity

    -

    Tidying Up the Integration Tests

    +

    Tidying Up the Integration Tests

    -

    We now have three sets of tests, all essentially pointing at the database: +

    + +We now have three sets of tests, all essentially pointing at the database: test_orm.py, test_repository.py, and test_uow.py. Should we throw any away?

    @@ -831,14 +1010,17 @@

    Tidying Up the Integration Tests

    -

    Wrap-Up

    +

    Wrap-Up

    -

    Hopefully we’ve convinced you that the Unit of Work pattern is useful, and +

    +Hopefully we’ve convinced you that the Unit of Work pattern is useful, and that the context manager is a really nice Pythonic way of visually grouping code into blocks that we want to happen atomically.

    -

    This pattern is so useful, in fact, that SQLAlchemy already uses a UoW +

    + +This pattern is so useful, in fact, that SQLAlchemy already uses a UoW in the shape of the Session object. The Session object in SQLAlchemy is the way that your application loads data from the database.

    @@ -848,7 +1030,8 @@

    Wrap-Up

    persisted together. Why do we go to the effort of abstracting away the SQLAlchemy session if it already implements the pattern we want?

    -

    Unit of Work pattern: the trade-offs discusses some of the trade-offs.

    +

    +Unit of Work pattern: the trade-offs discusses some of the trade-offs.

    @@ -869,7 +1052,9 @@

    Wrap-Up

  • We have a nice abstraction over the concept of atomic operations, and the context manager makes it easy to see, visually, what blocks of code are -grouped together atomically.

    +grouped together atomically. + +

  • We have explicit control over when a transaction starts and finishes, and our @@ -896,7 +1081,8 @@

    Wrap-Up

    We’ve made it look easy, but you have to think quite carefully about things like rollbacks, multithreading, and nested transactions. Perhaps just sticking to what Django or Flask-SQLAlchemy gives you will keep your life -simpler.

    +simpler. +

  • @@ -916,6 +1102,9 @@

    Wrap-Up

    Unit of Work Pattern Recap
    +
    +

    +
    The Unit of Work pattern is an abstraction around data integrity
    @@ -947,7 +1136,8 @@

    Wrap-Up

    -

    Lastly, we’re motivated again by the dependency inversion principle: our +

    +Lastly, we’re motivated again by the dependency inversion principle: our service layer depends on a thin abstraction, and we attach a concrete implementation at the outside edge of the system. This lines up nicely with SQLAlchemy’s own @@ -970,6 +1160,11 @@

    Wrap-Up

    +

    @@ -979,93 +1174,22 @@

    Wrap-Up

    -
    diff --git a/docs/book/chapter_07_aggregate.html b/book/chapter_07_aggregate.html similarity index 79% rename from docs/book/chapter_07_aggregate.html rename to book/chapter_07_aggregate.html index 9ae54ca..5b33ea0 100644 --- a/docs/book/chapter_07_aggregate.html +++ b/book/chapter_07_aggregate.html @@ -3,7 +3,7 @@ - + Aggregates and Consistency Boundaries + @@ -81,7 +183,7 @@
    -

    Aggregates and Consistency Boundaries

    +

    7: Aggregates and Consistency Boundaries

    -

    In this chapter, we’d like to revisit our domain model to talk about invariants +

    + + + +In this chapter, we’d like to revisit our domain model to talk about invariants and constraints, and see how our domain objects can maintain their own internal consistency, both conceptually and in persistent storage. We’ll discuss the concept of a consistency boundary and show how making it @@ -139,14 +245,14 @@

    Aggregates and Consistency Boundaries

    Table 1. Unit of Work pattern: the trade-offs
    -

    The code for this chapter is in the appendix_csvs branch -on GitHub:

    +

    The code for this chapter is in the chapter_07_aggregate branch +on GitHub:

    git clone https://github.com/cosmicpython/code.git
     cd code
    -git checkout appendix_csvs
    +git checkout chapter_07_aggregate
     # or to code along, checkout the previous chapter:
     git checkout chapter_06_uow
    @@ -156,18 +262,21 @@

    Aggregates and Consistency Boundaries

    -

    Why Not Just Run Everything in a Spreadsheet?

    +

    Why Not Just Run Everything in a Spreadsheet?

    -

    What’s the point of a domain model, anyway? What’s the fundamental problem +

    + +What’s the point of a domain model, anyway? What’s the fundamental problem we’re trying to address?

    Couldn’t we just run everything in a spreadsheet? Many of our users would be -delighted by that. Business users like spreadsheets because they’re simple, -familiar, and yet enormously powerful.

    +delighted by that. Business users like spreadsheets because +they’re simple, familiar, and yet enormously powerful.

    -

    In fact, an enormous number of business processes do operate by manually sending +

    +In fact, an enormous number of business processes do operate by manually sending spreadsheets back and forth over email. This "CSV over SMTP" architecture has low initial complexity but tends not to scale very well because it’s difficult to apply logic and maintain consistency.

    @@ -185,19 +294,23 @@

    Why Not Just Run Everythi

    -

    Invariants, Constraints, and Consistency

    +

    Invariants, Constraints, and Consistency

    -

    The two words are somewhat interchangeable, but a constraint is a +

    + +The two words are somewhat interchangeable, but a constraint is a rule that restricts the possible states our model can get into, while an invariant is defined a little more precisely as a condition that is always true.

    -

    If we were writing a hotel-booking system, we might have the constraint that double +

    +If we were writing a hotel-booking system, we might have the constraint that double bookings are not allowed. This supports the invariant that a room cannot have more than one booking for the same night.

    -

    Of course, sometimes we might need to temporarily bend the rules. Perhaps we +

    +Of course, sometimes we might need to temporarily bend the rules. Perhaps we need to shuffle the rooms around because of a VIP booking. While we’re moving bookings around in memory, we might be double booked, but our domain model should ensure that, when we’re finished, we end up in a final consistent state, @@ -218,22 +331,24 @@

    Invariants, Constraints, and Co

    -

    This is a business rule that imposes an invariant. The invariant is that an +

    +This is a business rule that imposes an invariant. The invariant is that an order line is allocated to either zero or one batch, but never more than one. We need to make sure that our code never accidentally calls Batch.allocate() on two different batches for the same line, and currently, there’s nothing there to explicitly stop us from doing that.

    -

    Invariants, Concurrency, and Locks

    +

    Invariants, Concurrency, and Locks

    -

    Let’s look at another one of our business rules:

    +

    +Let’s look at another one of our business rules:

    We can’t allocate to a batch if the available quantity is less than the - quantity of the order line.

    +quantity of the order line.

    @@ -241,7 +356,8 @@

    Invariants, Concurrency, and Locks

    -

    Here the constraint is that we can’t allocate more stock than is available to a +

    +Here the constraint is that we can’t allocate more stock than is available to a batch, so we never oversell stock by allocating two customers to the same physical cushion, for example. Every time we update the state of the system, our code needs to ensure that we don’t break the invariant, which is that the available @@ -253,13 +369,15 @@

    Invariants, Concurrency, and Locks

    -

    This gets much harder when we introduce the idea of concurrency. Suddenly we +

    +This gets much harder when we introduce the idea of concurrency. Suddenly we might be allocating stock for multiple order lines simultaneously. We might even be allocating order lines at the same time as processing changes to the batches themselves.

    -

    We usually solve this problem by applying locks to our database tables. This +

    +We usually solve this problem by applying locks to our database tables. This prevents two operations from happening simultaneously on the same row or same table.

    @@ -273,9 +391,12 @@

    Invariants, Concurrency, and Locks

    -

    What Is an Aggregate?

    +

    What Is an Aggregate?

    -

    OK, so if we can’t lock the whole database every time we want to allocate an +

    + + +OK, so if we can’t lock the whole database every time we want to allocate an order line, what should we do instead? We want to protect the invariants of our system but allow for the greatest degree of concurrency. Maintaining our invariants inevitably means preventing concurrent writes; if multiple users can @@ -288,7 +409,9 @@

    What Is an Aggregate?

    to be consistent with each other.

    -

    The Aggregate pattern is a design pattern from the DDD community that helps us +

    + +The Aggregate pattern is a design pattern from the DDD community that helps us to resolve this tension. An aggregate is just a domain object that contains other domain objects and lets us treat the whole collection as a single unit.

    @@ -297,7 +420,8 @@

    What Is an Aggregate?

    thing, and to call methods on the aggregate itself.

    -

    As a model gets more complex and grows more entity and value objects, +

    +As a model gets more complex and grows more entity and value objects, referencing each other in a tangled graph, it can be hard to keep track of who can modify what. Especially when we have collections in the model as we do (our batches are a collection), it’s a good idea to nominate some entities to be @@ -314,7 +438,8 @@

    What Is an Aggregate?

    basket to run in a single database transaction.

    -

    We don’t want to modify multiple baskets in a transaction, because there’s no +

    +We don’t want to modify multiple baskets in a transaction, because there’s no use case for changing the baskets of several customers at the same time. Each basket is a single consistency boundary responsible for maintaining its own invariants.

    @@ -323,7 +448,8 @@

    What Is an Aggregate?

    An AGGREGATE is a cluster of associated objects that we treat as a unit for the -purpose of data changes.

    +purpose of data changes. +

    @@ -352,9 +478,11 @@

    What Is an Aggregate?

    -

    Choosing an Aggregate

    +

    Choosing an Aggregate

    -

    What aggregate should we use for our system? The choice is somewhat arbitrary, +

    + +What aggregate should we use for our system? The choice is somewhat arbitrary, but it’s important. The aggregate will be the boundary where we make sure every operation ends in a consistent state. This helps us to reason about our software and prevent weird race issues. We want to draw a boundary around a @@ -362,7 +490,8 @@

    Choosing an Aggregate

    be consistent with one another, and we need to give this boundary a good name.

    -

    The object we’re manipulating under the covers is Batch. What do we call a +

    +The object we’re manipulating under the covers is Batch. What do we call a collection of batches? How should we divide all the batches in the system into discrete islands of consistency?

    @@ -374,7 +503,7 @@

    Choosing an Aggregate

    Neither of these concepts really satisfies us, though. We should be able to -allocate DEADLY-SPOONs and FLIMSY-DESKs at the same time, even if they’re in the +allocate DEADLY-SPOONs or FLIMSY-DESKs in one go, even if they’re not in the same warehouse or the same shipment. These concepts have the wrong granularity.

    @@ -384,11 +513,14 @@

    Choosing an Aggregate

    It’s an unwieldy name, though, so after some bikeshedding via SkuStock, Stock, -ProductStock, and so on, we decided to simply call it Product—after all, that was the first concept we came across in our exploration of the +ProductStock, and so on, we decided to simply call it Product—after all, +that was the first concept we came across in our exploration of the domain language back in [chapter_01_domain_model].

    -

    So the plan is this: when we want to allocate an order line, instead of +

    + +So the plan is this: when we want to allocate an order line, instead of Before: allocate against all batches using the domain service, where we look up all the Batch objects in the world and pass them to the allocate() domain service…​

    @@ -442,7 +574,9 @@

    Choosing an Aggregate

    -

    …​we’ll move to the world of After: ask Product to allocate against its batches, in which there is a new +

    + +…​we’ll move to the world of After: ask Product to allocate against its batches, in which there is a new Product object for the particular SKU of our order line, and it will be in charge of all the batches for that SKU, and we can call a .allocate() method on that instead.

    @@ -497,7 +631,8 @@

    Choosing an Aggregate

    -

    Let’s see how that looks in code form:

    +

    +Let’s see how that looks in code form:

    Our chosen aggregate, Product (src/allocation/domain/model.py)
    @@ -505,20 +640,17 @@

    Choosing an Aggregate

    class Product:
    -
         def __init__(self, sku: str, batches: List[Batch]):
             self.sku = sku  #(1)
             self.batches = batches  #(2)
     
         def allocate(self, line: OrderLine) -> str:  #(3)
             try:
    -            batch = next(
    -                b for b in sorted(self.batches) if b.can_allocate(line)
    -            )
    +            batch = next(b for b in sorted(self.batches) if b.can_allocate(line))
                 batch.allocate(line)
                 return batch.reference
             except StopIteration:
    -            raise OutOfStock(f'Out of stock for sku {line.sku}')
    + raise OutOfStock(f"Out of stock for sku {line.sku}")
    @@ -526,14 +658,15 @@

    Choosing an Aggregate

    1. -

      Product’s main identifier is the `sku.

      +

      Product's main identifier is the sku.

    2. -

      Our Product class holds a reference to a collection of batches for that SKU.

      +

      Our Product class holds a reference to a collection of batches for that SKU. +

    3. Finally, we can move the allocate() domain service to -be a method on the Product aggregate.

      +be a method on the Product aggregate.

    @@ -549,8 +682,8 @@

    Choosing an Aggregate

    Our allocation service doesn’t care about any of those things. This is the power of bounded contexts; the concept of a product in one app can be very different from another. - See the following sidebar for more - discussion. + See the following sidebar for more discussion. + @@ -559,12 +692,14 @@

    Choosing an Aggregate

    Aggregates, Bounded Contexts, and Microservices
    -

    One of the most important contributions from Evans and the DDD community +

    +One of the most important contributions from Evans and the DDD community is the concept of bounded contexts.

    -

    In essence, this was a reaction against attempts to capture entire businesses +

    +In essence, this was a reaction against attempts to capture entire businesses into a single model. The word customer means different things to people in sales, customer service, logistics, support, and so on. Attributes needed in one context are irrelevant in another; more perniciously, concepts @@ -575,7 +710,8 @@

    Choosing an Aggregate

    explicitly.

    -

    This concept translates very well to the world of microservices, where each +

    +This concept translates very well to the world of microservices, where each microservice is free to have its own concept of "customer" and its own rules for translating that to and from other microservices it integrates with.

    @@ -592,7 +728,8 @@

    Choosing an Aggregate

    aggregates low and their size manageable.

    -

    Once again, we find ourselves forced to say that we can’t give this issue +

    +Once again, we find ourselves forced to say that we can’t give this issue the treatment it deserves here, and we can only encourage you to read up on it elsewhere. The Fowler link at the start of this sidebar is a good starting point, and either (or indeed, any) DDD book will have a chapter or more on bounded contexts.

    @@ -601,9 +738,11 @@

    Choosing an Aggregate

    -

    One Aggregate = One Repository

    +

    One Aggregate = One Repository

    -

    Once you define certain entities to be aggregates, we need to apply the rule +

    + +Once you define certain entities to be aggregates, we need to apply the rule that they are the only entities that are publicly accessible to the outside world. In other words, the only repositories we are allowed should be repositories that return aggregates.

    @@ -623,7 +762,9 @@

    One Aggregate = One Repository

    -

    In our case, we’ll switch from BatchRepository to ProductRepository:

    +

    + +In our case, we’ll switch from BatchRepository to ProductRepository:

    Our new UoW and repository (unit_of_work.py and repository.py)
    @@ -637,11 +778,11 @@

    One Aggregate = One Repository

    class AbstractProductRepository(abc.ABC): - @abc.abstractmethod + @abc.abstractmethod def add(self, product): ... - @abc.abstractmethod + @abc.abstractmethod def get(self, sku) -> model.Product: ...
    @@ -649,7 +790,10 @@

    One Aggregate = One Repository

    -

    The ORM layer will need some tweaks so that the right batches automatically get +

    + + +The ORM layer will need some tweaks so that the right batches automatically get loaded and associated with Product objects. The nice thing is, the Repository pattern means we don’t have to worry about that yet. We can just use our FakeRepository and then feed through the new model into our service @@ -661,12 +805,12 @@

    One Aggregate = One Repository

    def add_batch(
    -        ref: str, sku: str, qty: int, eta: Optional[date],
    -        uow: unit_of_work.AbstractUnitOfWork
    +    ref: str, sku: str, qty: int, eta: Optional[date],
    +    uow: unit_of_work.AbstractUnitOfWork,
     ):
         with uow:
             product = uow.products.get(sku=sku)
    -        if product is None:
    +        if product is None:
                 product = model.Product(sku, batches=[])
                 uow.products.add(product)
             product.batches.append(model.Batch(ref, sku, qty, eta))
    @@ -674,14 +818,14 @@ 

    One Aggregate = One Repository

    def allocate( - orderid: str, sku: str, qty: int, - uow: unit_of_work.AbstractUnitOfWork + orderid: str, sku: str, qty: int, + uow: unit_of_work.AbstractUnitOfWork, ) -> str: line = OrderLine(orderid, sku, qty) with uow: product = uow.products.get(sku=line.sku) - if product is None: - raise InvalidSku(f'Invalid sku {line.sku}') + if product is None: + raise InvalidSku(f"Invalid sku {line.sku}") batchref = product.allocate(line) uow.commit() return batchref
    @@ -691,9 +835,11 @@

    One Aggregate = One Repository

    -

    What About Performance?

    +

    What About Performance?

    -

    We’ve mentioned a few times that we’re modeling with aggregates because we want +

    + +We’ve mentioned a few times that we’re modeling with aggregates because we want to have high-performance software, but here we are loading all the batches when we only need one. You might expect that to be inefficient, but there are a few reasons why we’re comfortable here.

    @@ -727,7 +873,8 @@

    What About Performance?

    Exercise for the Reader
    -

    You’ve just seen the main top layers of the code, so this shouldn’t be too hard, +

    +You’ve just seen the main top layers of the code, so this shouldn’t be too hard, but we’d like you to implement the Product aggregate starting from Batch, just as we did.

    @@ -738,14 +885,17 @@

    What About Performance?

    talk to each other, which we hope will be instructive.

    -

    You’ll find the code on GitHub. We’ve put in a "cheating" implementation in the delegates to the existing +

    You’ll find the code on GitHub. +We’ve put in a "cheating" implementation in the delegates to the existing allocate() function, so you should be able to evolve that toward the real thing.

    -

    We’ve marked a couple of tests with @pytest.skip(). After you’ve read the rest of this chapter, come back to these tests to have a go -at implementing version numbers. Bonus points if you can get SQLAlchemy to -do them for you by magic!

    +

    +We’ve marked a couple of tests with @pytest.skip(). After you’ve read the +rest of this chapter, come back to these tests to have a go at implementing +version numbers. Bonus points if you can get SQLAlchemy to do them for you by +magic!

    @@ -759,9 +909,12 @@

    What About Performance?

    -

    Optimistic Concurrency with Version Numbers

    +

    Optimistic Concurrency with Version Numbers

    -

    We have our new aggregate, so we’ve solved the conceptual problem of choosing +

    + + +We have our new aggregate, so we’ve solved the conceptual problem of choosing an object to be in charge of consistency boundaries. Let’s now spend a little time talking about how to enforce data integrity at the database level.

    @@ -772,18 +925,24 @@

    Optimistic Concurrency wit
    Note
    -This section has a lot of implementation details; for example, some of it is Postgres-specific. But more generally, we’re showing one way of managing concurrency issues, but it is just one approach. Real requirements in this area vary a lot from project to project. You - shouldn’t expect to be able to copy and paste code from here into production. +This section has a lot of implementation details; for example, some of it + is Postgres-specific. But more generally, we’re showing one way of managing + concurrency issues, but it is just one approach. Real requirements in this + area vary a lot from project to project. You shouldn’t expect to be able to + copy and paste code from here into production. +

    -

    We don’t want to hold a lock over the entire batches table, but how will we +

    +We don’t want to hold a lock over the entire batches table, but how will we implement holding a lock over just the rows for a particular SKU?

    -

    One answer is to have a single attribute on the Product model that acts as a marker for +

    +One answer is to have a single attribute on the Product model that acts as a marker for the whole state change being complete and to use it as the single resource that concurrent workers can fight over. If two transactions read the state of the world for batches at the same time, and both want to update @@ -792,7 +951,9 @@

    Optimistic Concurrency wit can win and the world stays consistent.

    -

    Sequence diagram: two transactions attempting a concurrent update on Product illustrates two concurrent +

    + +Sequence diagram: two transactions attempting a concurrent update on Product illustrates two concurrent transactions doing their read operations at the same time, so they see a Product with, for example, version=3. They both call Product.allocate() in order to modify a state. But we set up our database integrity @@ -810,6 +971,7 @@

    Optimistic Concurrency wit could achieve the same thing by setting the Postgres transaction isolation level to SERIALIZABLE, but that often comes at a severe performance cost. Version numbers also make implicit concepts explicit. + @@ -818,7 +980,7 @@

    Optimistic Concurrency wit
    apwp 0704
    -
    Figure 4. Sequence diagram: two transactions attempting a concurrent update on Product
    +
    Figure 4. Sequence diagram: two transactions attempting a concurrent update on Product

    @@ -859,7 +1021,10 @@

    Optimistic Concurrency wit notice if there is a problem.

    -

    Pessimistic concurrency control works under the assumption that two users +

    + + +Pessimistic concurrency control works under the assumption that two users are going to cause conflicts, and we want to prevent conflicts in all cases, so we lock everything just to be safe. In our example, that would mean locking the whole batches table, or using SELECT FOR UPDATE—we’re pretending @@ -867,13 +1032,15 @@

    Optimistic Concurrency wit want to do some evaluations and measurements of your own.

    -

    With pessimistic locking, you don’t need to think about handling failures +

    +With pessimistic locking, you don’t need to think about handling failures because the database will prevent them for you (although you do need to think about deadlocks). With optimistic locking, you need to explicitly handle the possibility of failures in the (hopefully unlikely) case of a clash.

    -

    The usual way to handle a failure is to retry the failed operation from the +

    +The usual way to handle a failure is to retry the failed operation from the beginning. Imagine we have two customers, Harry and Bob, and each submits an order for SHINY-TABLE. Both threads load the product at version 1 and allocate stock. The database prevents the concurrent update, and Bob’s order fails with @@ -888,9 +1055,11 @@

    Optimistic Concurrency wit

    -

    Implementation Options for Version Numbers

    +

    Implementation Options for Version Numbers

    -

    There are essentially three options for implementing version numbers:

    +

    + +There are essentially three options for implementing version numbers:

      @@ -931,7 +1100,6 @@

      Implementation Options for
      class Product:
      -
           def __init__(self, sku: str, batches: List[Batch], version_number: int = 0):  #(1)
               self.sku = sku
               self.batches = batches
      @@ -939,14 +1107,12 @@ 

      Implementation Options for def allocate(self, line: OrderLine) -> str: try: - batch = next( - b for b in sorted(self.batches) if b.can_allocate(line) - ) + batch = next(b for b in sorted(self.batches) if b.can_allocate(line)) batch.allocate(line) self.version_number += 1 #(1) return batch.reference except StopIteration: - raise OutOfStock(f'Out of stock for sku {line.sku}')

      + raise OutOfStock(f"Out of stock for sku {line.sku}")

    @@ -971,6 +1137,9 @@

    Implementation Options for Product aggregate. The version number is a simple, human-comprehensible way to model a thing that changes on every write, but it could equally be a random UUID every time. + + + @@ -978,14 +1147,21 @@

    Implementation Options for

    -

    Testing for Our Data Integrity Rules

    +

    Testing for Our Data Integrity Rules

    -

    Now to make sure we can get the behavior we want: if we have two +

    + + +Now to make sure we can get the behavior we want: if we have two concurrent attempts to do allocation against the same Product, one of them should fail, because they can’t both update the version number.

    -

    First, let’s simulate a "slow" transaction using a function that does +

    + + + +First, let’s simulate a "slow" transaction using a function that does allocation and then does an explicit sleep:[2]

    @@ -1002,14 +1178,16 @@

    Testing for Our Data Integrity Ru time.sleep(0.2) uow.commit() except Exception as e: - print(traceback.format_exc()) + print(traceback.format_exc()) exceptions.append(e)

    -

    Then we have our test invoke this slow allocation twice, concurrently, using +

    + +Then we have our test invoke this slow allocation twice, concurrently, using threads:

    @@ -1020,7 +1198,7 @@

    Testing for Our Data Integrity Ru
    def test_concurrent_updates_to_version_are_not_allowed(postgres_session_factory):
         sku, batch = random_sku(), random_batchref()
         session = postgres_session_factory()
    -    insert_batch(session, batch, sku, 100, eta=None, product_version=1)
    +    insert_batch(session, batch, sku, 100, eta=None, product_version=1)
         session.commit()
     
         order1, order2 = random_orderid(1), random_orderid(2)
    @@ -1040,18 +1218,18 @@ 

    Testing for Our Data Integrity Ru ) assert version == 2 #(2) [exception] = exceptions - assert 'could not serialize access due to concurrent update' in str(exception) #(3) + assert "could not serialize access due to concurrent update" in str(exception) #(3) - orders = list(session.execute( + orders = session.execute( "SELECT orderid FROM allocations" " JOIN batches ON allocations.batch_id = batches.id" " JOIN order_lines ON allocations.orderline_id = order_lines.id" " WHERE order_lines.sku=:sku", dict(sku=sku), - )) - assert len(orders) == 1 #(4) + ) + assert orders.rowcount == 1 #(4) with unit_of_work.SqlAlchemyUnitOfWork() as uow: - uow.session.execute('select 1')

    + uow.session.execute("select 1")

    @@ -1074,9 +1252,11 @@

    Testing for Our Data Integrity Ru

    -

    Enforcing Concurrency Rules by Using Database Transaction Isolation Levels

    +

    Enforcing Concurrency Rules by Using Database Transaction Isolation Levels

    -

    To get the test to pass as it is, we can set the transaction isolation level +

    + +To get the test to pass as it is, we can set the transaction isolation level on our session:

    @@ -1084,10 +1264,12 @@

    -
    DEFAULT_SESSION_FACTORY = sessionmaker(bind=create_engine(
    -    config.get_postgres_uri(),
    -    isolation_level="REPEATABLE READ",
    -))
    +
    DEFAULT_SESSION_FACTORY = sessionmaker(
    +    bind=create_engine(
    +        config.get_postgres_uri(),
    +        isolation_level="REPEATABLE READ",
    +    )
    +)

    @@ -1100,21 +1282,27 @@

    Transaction isolation levels are tricky stuff, so it’s worth spending time -understanding the Postgres documentation.[3] + understanding the Postgres documentation.[3] + +

    -

    Pessimistic Concurrency Control Example: SELECT FOR UPDATE

    +

    Pessimistic Concurrency Control Example: SELECT FOR UPDATE

    -

    There are multiple ways to approach this, but we’ll show one. SELECT FOR UPDATE +

    + + +There are multiple ways to approach this, but we’ll show one. SELECT FOR UPDATE produces different behavior; two concurrent transactions will not be allowed to do a read on the same rows at the same time:

    -

    SELECT FOR UPDATE is a way of picking a row or rows to use as a lock +

    +SELECT FOR UPDATE is a way of picking a row or rows to use as a lock (although those rows don’t have to be the ones you update). If two transactions both try to SELECT FOR UPDATE a row at the same time, one will win, and the other will wait until the lock is released. So this is an example @@ -1130,10 +1318,12 @@

    Pessimistic
        def get(self, sku):
    -        return self.session.query(model.Product) \
    -                           .filter_by(sku=sku) \
    -                           .with_for_update() \
    -                           .first()
    + return ( + self.session.query(model.Product) + .filter_by(sku=sku) + .with_for_update() + .first() + )

    @@ -1155,11 +1345,15 @@

    Pessimistic

    -

    Some people refer to this as the "read-modify-write" failure mode. +

    + +Some people refer to this as the "read-modify-write" failure mode. Read "PostgreSQL Anti-Patterns: Read-Modify-Write Cycles" for a good overview.

    -

    We don’t really have time to discuss all the trade-offs between REPEATABLE READ +

    + +We don’t really have time to discuss all the trade-offs between REPEATABLE READ and SELECT FOR UPDATE, or optimistic versus pessimistic locking in general. But if you have a test like the one we’ve shown, you can specify the behavior you want and see how it changes. You can also use the test as a basis for @@ -1168,9 +1362,10 @@

    Pessimistic

    -

    Wrap-Up

    +

    Wrap-Up

    -

    Specific choices around concurrency control vary a lot based on business +

    +Specific choices around concurrency control vary a lot based on business circumstances and storage technology choices, but we’d like to bring this chapter back to the conceptual idea of an aggregate: we explicitly model an object as being the main entrypoint to some subset of our model, and as being in @@ -1178,14 +1373,18 @@

    Wrap-Up

    those objects.

    -

    Choosing the right aggregate is key, and it’s a decision you may revisit +

    + + +Choosing the right aggregate is key, and it’s a decision you may revisit over time. You can read more about it in multiple DDD books. We also recommend these three online papers on effective aggregate design by Vaughn Vernon (the "red book" author).

    -

    Aggregates: the trade-offs has some thoughts on the trade-offs of implementing the Aggregate pattern.

    +

    +Aggregates: the trade-offs has some thoughts on the trade-offs of implementing the Aggregate pattern.

    @@ -1212,7 +1411,8 @@

    Wrap-Up

  • Modeling our operations around explicit consistency boundaries helps us avoid -performance problems with our ORM.

    +performance problems with our ORM. +

  • Putting the aggregate in sole charge of state changes to its subsidiary models @@ -1242,6 +1442,9 @@

    Wrap-Up

    Aggregates and Consistency Boundaries Recap
    +
    +

    +
    Aggregates are your entrypoints into the domain model
    @@ -1262,7 +1465,8 @@

    Wrap-Up

    When thinking about implementing these consistency checks, we end up thinking about transactions and locks. Choosing the right aggregate is about performance as well as conceptual -organization of your domain.

    +organization of your domain. +

    @@ -1270,9 +1474,10 @@

    Wrap-Up

    -

    Part I Recap

    +

    Part I Recap

    -

    Do you remember A component diagram for our app at the end of Part I, the diagram we showed at the +

    +Do you remember A component diagram for our app at the end of Part I, the diagram we showed at the beginning of [part1] to preview where we were heading?

    @@ -1298,10 +1503,13 @@

    Part I Recap

    big ball of mud.

    -

    By applying the dependency inversion principle, and by using ports-and-adapters-inspired patterns like Repository and Unit of Work, we’ve made it possible to -do TDD in both high gear and low gear and to maintain a healthy test pyramid. -We can test our system edge to edge, and the need for integration and -end-to-end tests is kept to a minimum.

    +

    + +By applying the dependency inversion principle, and by using +ports-and-adapters-inspired patterns like Repository and Unit of Work, we’ve +made it possible to do TDD in both high gear and low gear and to maintain a +healthy test pyramid. We can test our system edge to edge, and the need for +integration and end-to-end tests is kept to a minimum.

    Lastly, we’ve talked about the idea of consistency boundaries. We don’t want to @@ -1327,6 +1535,8 @@

    Part I Recap

    wrapper around a database and isn’t likely to be anything more than that in the foreseeable future, you don’t need these patterns. Go ahead and use Django, and save yourself a lot of bother. + +
  • Table 1. Aggregates: the trade-offs
    @@ -1339,6 +1549,11 @@

    Part I Recap

    +

    @@ -1349,98 +1564,27 @@

    Part I Recap

    2. time.sleep() works well in our use case, but it’s not the most reliable or efficient way to reproduce concurrency bugs. Consider using semaphores or similar synchronization primitives shared between your threads to get better guarantees of behavior.
    -3. If you’re not using Postgres, you’ll need to read different documentation. Annoyingly, different databases all have quite different definitions. Oracle’s SERIALIZABLE is equivalent to Postgres’s REPEATABLE READ, for example. +3. If you’re not using Postgres, you’ll need to read different documentation. Annoyingly, different databases all have quite different definitions. Oracle’s SERIALIZABLE is equivalent to Postgres’s REPEATABLE READ, for example.
    -
    diff --git a/docs/book/chapter_08_events_and_message_bus.html b/book/chapter_08_events_and_message_bus.html similarity index 80% rename from docs/book/chapter_08_events_and_message_bus.html rename to book/chapter_08_events_and_message_bus.html index 9c53e82..c49bf9c 100644 --- a/docs/book/chapter_08_events_and_message_bus.html +++ b/book/chapter_08_events_and_message_bus.html @@ -3,7 +3,7 @@ - + Events and the Message Bus + @@ -81,7 +183,7 @@
    -

    Events and the Message Bus

    +

    8: Events and the Message Bus

    -

    So far we’ve spent a lot of time and energy on a simple problem that we could +

    +So far we’ve spent a lot of time and energy on a simple problem that we could easily have solved with Django. You might be asking if the increased testability and expressiveness are really worth all the effort.

    @@ -136,7 +239,11 @@

    Events and the Message Bus

    why it’s exactly this kind of decision that leads us to the Big Ball of Mud.

    -

    Then we’ll show how to use the Domain Events pattern to separate side effects from our +

    + + + +Then we’ll show how to use the Domain Events pattern to separate side effects from our use cases, and how to use a simple Message Bus pattern for triggering behavior based on those events. We’ll show a few options for creating those events and how to pass them to the message bus, and finally we’ll show @@ -174,40 +281,44 @@

    Events and the Message Bus

    -

    Avoiding Making a Mess

    +

    Avoiding Making a Mess

    -

    So. Email alerts when we run out of stock. When we have new requirements like ones that really have nothing to do with the core domain, it’s all too easy to +

    + + +So. Email alerts when we run out of stock. When we have new requirements like ones that really have nothing to do with the core domain, it’s all too easy to start dumping these things into our web controllers.

    -

    First, Let’s Avoid Making a Mess of Our Web Controllers

    +

    First, Let’s Avoid Making a Mess of Our Web Controllers

    -

    As a one-off hack, this might be OK:

    +

    +As a one-off hack, this might be OK:

    Just whack it in the endpoint—what could go wrong? (src/allocation/entrypoints/flask_app.py)
    @@ -219,9 +330,11 @@

    First, Let’

    -

    And Let’s Not Make a Mess of Our Model Either

    +

    And Let’s Not Make a Mess of Our Model Either

    -

    Assuming we don’t want to put this code into our web controllers, because +

    + +Assuming we don’t want to put this code into our web controllers, because we want them to be as thin as possible, we may look at putting it right at the source, in the model:

    @@ -232,13 +345,11 @@

    And Let’s Not Make
        def allocate(self, line: OrderLine) -> str:
             try:
    -            batch = next(
    -                b for b in sorted(self.batches) if b.can_allocate(line)
    -            )
    +            batch = next(b for b in sorted(self.batches) if b.can_allocate(line))
                 #...
             except StopIteration:
    -            email.send_mail('stock@made.com', f'Out of stock for {line.sku}')
    -            raise OutOfStock(f'Out of stock for sku {line.sku}')
    + email.send_mail("stock@made.com", f"Out of stock for {line.sku}") + raise OutOfStock(f"Out of stock for sku {line.sku}")

    @@ -252,17 +363,13 @@

    And Let’s Not Make of our system. What we’d like is to keep our domain model focused on the rule "You can’t allocate more stuff than is actually available."

    -
    -

    The domain model’s job is to know that we’re out of stock, but the -responsibility of sending an alert belongs elsewhere. We should be able to turn -this feature on or off, or to switch to SMS notifications instead, without -needing to change the rules of our domain model.

    -
    -

    Or the Service Layer!

    +

    Or the Service Layer!

    -

    The requirement "Try to allocate some stock, and send an email if it fails" is +

    + +The requirement "Try to allocate some stock, and send an email if it fails" is an example of workflow orchestration: it’s a set of steps that the system has to follow to achieve a goal.

    @@ -276,38 +383,43 @@

    Or the Service Layer!

    def allocate(
    -        orderid: str, sku: str, qty: int,
    -        uow: unit_of_work.AbstractUnitOfWork
    +    orderid: str, sku: str, qty: int,
    +    uow: unit_of_work.AbstractUnitOfWork,
     ) -> str:
         line = OrderLine(orderid, sku, qty)
         with uow:
             product = uow.products.get(sku=line.sku)
    -        if product is None:
    -            raise InvalidSku(f'Invalid sku {line.sku}')
    +        if product is None:
    +            raise InvalidSku(f"Invalid sku {line.sku}")
             try:
                 batchref = product.allocate(line)
                 uow.commit()
                 return batchref
             except model.OutOfStock:
    -            email.send_mail('stock@made.com', f'Out of stock for {line.sku}')
    +            email.send_mail("stock@made.com", f"Out of stock for {line.sku}")
                 raise
    -

    Catching an exception and reraising it? It could be worse, but it’s +

    + +Catching an exception and reraising it? It could be worse, but it’s definitely making us unhappy. Why is it so hard to find a suitable home for this code?

    -

    Single Responsibility Principle

    +

    Single Responsibility Principle

    -

    Really, this is a violation of the single responsibility principle (SRP).[1] +

    + +Really, this is a violation of the single responsibility principle (SRP).[1] Our use case is allocation. Our endpoint, service function, and domain methods -are all called allocate, not allocate_and_send_mail_if_out_of_stock.

    +are all called allocate, not +allocate_and_send_mail_if_out_of_stock.

    @@ -328,9 +440,11 @@

    Single Responsibility Principle

    allocate() function, because that’s clearly a separate responsibility.

    -

    To solve the problem, we’re going to split the orchestration -into separate steps so that the different concerns don’t get tangled up.[2] The -domain model’s job is to know that we’re out of stock, but the responsibility +

    + +To solve the problem, we’re going to split the orchestration +into separate steps so that the different concerns don’t get tangled up.[2] +The domain model’s job is to know that we’re out of stock, but the responsibility of sending an alert belongs elsewhere. We should be able to turn this feature on or off, or to switch to SMS notifications instead, without needing to change the rules of our domain model.

    @@ -343,22 +457,27 @@

    Single Responsibility Principle

    -

    All Aboard the Message Bus!

    +

    All Aboard the Message Bus!

    The patterns we’re going to introduce here are Domain Events and the Message Bus. -We can implement them in a few ways, so we’ll show a couple before settling on the one we like most.

    +We can implement them in a few ways, so we’ll show a couple before settling on +the one we like most.

    -

    The Model Records Events

    +

    The Model Records Events

    -

    First, rather than being concerned about emails, our model will be in charge of -recording events—facts about things that have happened. We’ll use a message bus to respond to events and invoke a new operation.

    +

    +First, rather than being concerned about emails, our model will be in charge of +recording events—facts about things that have happened. We’ll use a message +bus to respond to events and invoke a new operation.

    -

    Events Are Simple Dataclasses

    +

    Events Are Simple Dataclasses

    -

    An event is a kind of value object. Events don’t have any behavior, because +

    + +An event is a kind of value object. Events don’t have any behavior, because they’re pure data structures. We always name events in the language of the domain, and we think of them as part of our domain model.

    @@ -374,9 +493,11 @@

    Events Are Simple Dataclasses

    from dataclasses import dataclass
     
    +
     class Event:  #(1)
         pass
     
    +
     @dataclass
     class OutOfStock(Event):  #(2)
         sku: str
    @@ -398,12 +519,15 @@

    Events Are Simple Dataclasses

    -

    The Model Raises Events

    +

    The Model Raises Events

    -

    When our domain model records a fact that happened, we say it raises an event.

    +

    + +When our domain model records a fact that happened, we say it raises an event.

    -

    Here’s what it will look like from the outside; if we ask Product to allocate +

    +Here’s what it will look like from the outside; if we ask Product to allocate but it can’t, it should raise an event:

    @@ -412,13 +536,13 @@

    The Model Raises Events

    def test_records_out_of_stock_event_if_cannot_allocate():
    -    batch = Batch('batch1', 'SMALL-FORK', 10, eta=today)
    +    batch = Batch("batch1", "SMALL-FORK", 10, eta=today)
         product = Product(sku="SMALL-FORK", batches=[batch])
    -    product.allocate(OrderLine('order1', 'SMALL-FORK', 10))
    +    product.allocate(OrderLine("order1", "SMALL-FORK", 10))
     
    -    allocation = product.allocate(OrderLine('order2', 'SMALL-FORK', 1))
    +    allocation = product.allocate(OrderLine("order2", "SMALL-FORK", 1))
         assert product.events[-1] == events.OutOfStock(sku="SMALL-FORK")  #(1)
    -    assert allocation is None
    + assert allocation is None
    @@ -440,7 +564,6 @@

    The Model Raises Events

    class Product:
    -
         def __init__(self, sku: str, batches: List[Batch], version_number: int = 0):
             self.sku = sku
             self.batches = batches
    @@ -452,8 +575,8 @@ 

    The Model Raises Events

    #... except StopIteration: self.events.append(events.OutOfStock(line.sku)) #(2) - # raise OutOfStock(f'Out of stock for sku {line.sku}') #(3) - return None
    + # raise OutOfStock(f"Out of stock for sku {line.sku}") #(3) + return None
    @@ -486,15 +609,20 @@

    The Model Raises Events

    events, don’t raise exceptions to describe the same domain concept. As you’ll see later when we handle events in the Unit of Work pattern, it’s confusing to have to reason about events and exceptions together. + +
    -

    The Message Bus Maps Events to Handlers

    +

    The Message Bus Maps Events to Handlers

    -

    A message bus basically says, "When I see this event, I should invoke the following +

    + + +A message bus basically says, "When I see this event, I should invoke the following handler function." In other words, it’s a simple publish-subscribe system. Handlers are subscribed to receive events, which we publish to the bus. It sounds harder than it is, and we usually implement it with a dict:

    @@ -511,14 +639,13 @@

    The Message Bus Maps Events to def send_out_of_stock_notification(event: events.OutOfStock): email.send_mail( - 'stock@made.com', - f'Out of stock for {event.sku}', + "stock@made.com", + f"Out of stock for {event.sku}", ) HANDLERS = { events.OutOfStock: [send_out_of_stock_notification], - } # type: Dict[Type[events.Event], List[Callable]]

    @@ -532,12 +659,12 @@

    The Message Bus Maps Events to Note that the message bus as implemented doesn’t give us concurrency because - only one handler will run at a time. - Our objective isn’t to support parallel threads but to separate - tasks conceptually, and to keep each UoW as small as possible. - This helps us to understand the codebase because the "recipe" for how to - run each use case is written in a single place. - See the following sidebar. + only one handler will run at a time. Our objective isn’t to support + parallel threads but to separate tasks conceptually, and to keep each UoW + as small as possible. This helps us to understand the codebase because the + "recipe" for how to run each use case is written in a single place. See the + following sidebar. + @@ -546,13 +673,15 @@

    The Message Bus Maps Events to
    Is This Like Celery?
    -

    Celery is a popular tool in the Python world for deferring self-contained +

    +Celery is a popular tool in the Python world for deferring self-contained chunks of work to an asynchronous task queue. The message bus we’re presenting here is very different, so the short answer to the above question is no; our message bus -has more in common with a Node.js app, a UI event loop, or an actor framework.

    +has more in common with an Express.js app, a UI event loop, or an actor framework.

    -

    If you do have a requirement for moving work off the main thread, you +

    +If you do have a requirement for moving work off the main thread, you can still use our event-based metaphors, but we suggest you use external events for that. There’s more discussion in [chapter_11_external_events_tradeoffs], but essentially, if you @@ -576,9 +705,14 @@

    The Message Bus Maps Events to

    -

    Option 1: The Service Layer Takes Events from the Model and Puts Them on the Message Bus

    +

    Option 1: The Service Layer Takes Events from the Model and Puts Them on the Message Bus

    -

    Our domain model raises events, and our message bus will call the right +

    + + + + +Our domain model raises events, and our message bus will call the right handlers whenever an event happens. Now all we need is to connect the two. We need something to catch events from the model and pass them to the message bus—​the publishing step.

    @@ -595,14 +729,14 @@

    ... def allocate( - orderid: str, sku: str, qty: int, - uow: unit_of_work.AbstractUnitOfWork + orderid: str, sku: str, qty: int, + uow: unit_of_work.AbstractUnitOfWork, ) -> str: line = OrderLine(orderid, sku, qty) with uow: product = uow.products.get(sku=line.sku) - if product is None: - raise InvalidSku(f'Invalid sku {line.sku}') + if product is None: + raise InvalidSku(f"Invalid sku {line.sku}") try: #(1) batchref = product.allocate(line) uow.commit() @@ -634,9 +768,12 @@

    -

    Option 2: The Service Layer Raises Its Own Events

    +

    Option 2: The Service Layer Raises Its Own Events

    -

    Another variant on this that we’ve used is to have the service layer +

    + + +Another variant on this that we’ve used is to have the service layer in charge of creating and raising events directly, rather than having them raised by the domain model:

    @@ -646,18 +783,18 @@

    Option 2: The Service

    -

    Option 3: The UoW Publishes Events to the Message Bus

    +

    Option 3: The UoW Publishes Events to the Message Bus

    -

    The UoW already has a try/finally, and it knows about all the aggregates +

    + + +The UoW already has a try/finally, and it knows about all the aggregates currently in play because it provides access to the repository. So it’s a good place to spot events and pass them to the message bus:

    @@ -706,7 +846,7 @@

    Option 3: The UoW event = product.events.pop(0) messagebus.handle(event) - @abc.abstractmethod + @abc.abstractmethod def _commit(self): raise NotImplementedError @@ -733,7 +873,9 @@

    Option 3: The UoW
  • That relies on the repository keeping track of aggregates that have been loaded -using a new attribute, .seen, as you’ll see in the next listing.

    +using a new attribute, .seen, as you’ll see in the next listing. + +

  • @@ -756,7 +898,6 @@

    Option 3: The UoW
    class AbstractRepository(abc.ABC):
    -
         def __init__(self):
             self.seen = set()  # type: Set[model.Product]  #(1)
     
    @@ -770,18 +911,16 @@ 

    Option 3: The UoW self.seen.add(product) return product - @abc.abstractmethod + @abc.abstractmethod def _add(self, product: model.Product): #(2) raise NotImplementedError - @abc.abstractmethod #(3) + @abc.abstractmethod #(3) def _get(self, sku) -> model.Product: raise NotImplementedError - class SqlAlchemyRepository(AbstractRepository): - def __init__(self, session): super().__init__() self.session = session @@ -801,7 +940,8 @@

    Option 3: The UoW

    For the UoW to be able to publish new events, it needs to be able to ask the repository for which Product objects have been used during this session. We use a set called .seen to store them. That means our implementations -need to call super().__init__().

    +need to call super().__init__(). +

  • The parent add() method adds things to .seen, and now requires subclasses @@ -831,7 +971,8 @@

    Option 3: The UoW

    After the UoW and repository collaborate in this way to automatically keep track of live objects and process their events, the service layer can be -totally free of event-handling concerns:

    +totally free of event-handling concerns: +

    Service layer is clean again (src/allocation/service_layer/services.py)
    @@ -839,14 +980,14 @@

    Option 3: The UoW
    def allocate(
    -        orderid: str, sku: str, qty: int,
    -        uow: unit_of_work.AbstractUnitOfWork
    +    orderid: str, sku: str, qty: int,
    +    uow: unit_of_work.AbstractUnitOfWork,
     ) -> str:
         line = OrderLine(orderid, sku, qty)
         with uow:
             product = uow.products.get(sku=line.sku)
    -        if product is None:
    -            raise InvalidSku(f'Invalid sku {line.sku}')
    +        if product is None:
    +            raise InvalidSku(f"Invalid sku {line.sku}")
             batchref = product.allocate(line)
             uow.commit()
             return batchref
    @@ -855,7 +996,11 @@

    Option 3: The UoW

    -

    We do also have to remember to change the fakes in the service layer and make them +

    + + + +We do also have to remember to change the fakes in the service layer and make them call super() in the right places, and to implement underscorey methods, but the changes are minimal:

    @@ -865,7 +1010,6 @@

    Option 3: The UoW
    class FakeRepository(repository.AbstractRepository):
    -
         def __init__(self, products):
             super().__init__()
             self._products = set(products)
    @@ -874,7 +1018,7 @@ 

    Option 3: The UoW self._products.add(product) def _get(self, sku): - return next((p for p in self._products if p.sku == sku), None) + return next((p for p in self._products if p.sku == sku), None) ... @@ -882,7 +1026,7 @@

    Option 3: The UoW ... def _commit(self): - self.committed = True

    + self.committed = True

  • @@ -891,7 +1035,11 @@

    Option 3: The UoW
    Exercise for the Reader
    -

    Are you finding all those ._add() and ._commit() methods "super-gross," in +

    + + + +Are you finding all those ._add() and ._commit() methods "super-gross," in the words of our beloved tech reviewer Hynek? Does it "make you want to beat Harry around the head with a plushie snake"? Hey, our code listings are only meant to be examples, not the perfect solution! Why not go see if you @@ -935,11 +1083,15 @@

    Option 3: The UoW

    -

    See if you can apply a similar pattern to our UoW class in -order to get rid of those Java-y _commit() methods too. You can find the code on GitHub.

    +

    +See if you can apply a similar pattern to our UoW class in +order to get rid of those Java-y _commit() methods too. You can find the code +on GitHub.

    -

    Switching all the ABCs to typing.Protocol is a good way to force yourself to avoid using inheritance. Let us know if you come up with something nice!

    +

    +Switching all the ABCs to typing.Protocol is a good way to force yourself to +avoid using inheritance. Let us know if you come up with something nice!

    @@ -952,7 +1104,7 @@

    Option 3: The UoW
    -

    Wrap-Up

    +

    Wrap-Up

    Domain events give us a way to handle workflows in our system. We often find, listening to our domain experts, that they express requirements in a causal or @@ -965,7 +1117,9 @@

    Wrap-Up

    us make our code more testable and observable, and it helps isolate concerns.

    -

    And Domain events: the trade-offs shows the trade-offs as we +

    + +And Domain events: the trade-offs shows the trade-offs as we see them.

    @@ -1011,7 +1165,8 @@

    Wrap-Up

    meaning your service-layer function doesn’t finish until all the handlers for any events are finished. That could cause unexpected performance problems in your web endpoints -(adding asynchronous processing is possible but makes things even more confusing).

    +(adding asynchronous processing is possible but makes things even more confusing). +

  • More generally, event-driven workflows can be confusing because after things @@ -1020,7 +1175,9 @@

    Wrap-Up

  • You also open yourself up to the possibility of circular dependencies between your -event handlers, and infinite loops.

    +event handlers, and infinite loops. + +

  • @@ -1028,7 +1185,8 @@

    Wrap-Up

    -

    Events are useful for more than just sending email, though. In [chapter_07_aggregate] we +

    +Events are useful for more than just sending email, though. In [chapter_07_aggregate] we spent a lot of time convincing you that you should define aggregates, or boundaries where we guarantee consistency. People often ask, "What should I do if I need to change multiple aggregates as part of a request?" Now @@ -1043,6 +1201,10 @@

    Wrap-Up

    Domain Events and the Message Bus Recap
    +
    +

    +

    +
    Events can help with the single responsibility principle
    @@ -1063,14 +1225,16 @@

    Wrap-Up

    The simplest way to start using events in your system is to raise them from handlers by calling bus.handle(some_new_event) after you commit your -unit of work.

    +unit of work. +

    Option 2: Domain model raises events, service layer passes them to message bus

    The logic about when to raise an event really should live with the model, so we can improve our system’s design and testability by raising events from the domain model. It’s easy for our handlers to collect events off the model -objects after commit and pass them to the bus.

    +objects after commit and pass them to the bus. +

    Option 3: UoW collects events from aggregates and passes them to message bus
    @@ -1078,7 +1242,9 @@

    Wrap-Up

    can tidy up by making our unit of work responsible for raising events that were raised by loaded objects. This is the most complex design and might rely on ORM magic, but it’s clean -and easy to use once it’s set up.

    +and easy to use once it’s set up. + +

    @@ -1091,6 +1257,11 @@

    Wrap-Up

    +

    @@ -1098,98 +1269,27 @@

    Wrap-Up

    1. This principle is the S in SOLID.
    -2. Our tech reviewer Ed Jung likes to say that the move from imperative to event-based flow control changes what used to be orchestration into choreography. +2. Our tech reviewer Ed Jung likes to say that when you change from imperative flow control to event-based flow control, you’re changing orchestration into choreography.
    -
    diff --git a/docs/book/chapter_09_all_messagebus.html b/book/chapter_09_all_messagebus.html similarity index 80% rename from docs/book/chapter_09_all_messagebus.html rename to book/chapter_09_all_messagebus.html index 014b7bc..6d7ee57 100644 --- a/docs/book/chapter_09_all_messagebus.html +++ b/book/chapter_09_all_messagebus.html @@ -3,7 +3,7 @@ - + Going to Town on the Message Bus + @@ -81,7 +183,7 @@
    -

    Going to Town on the Message Bus

    +

    9: Going to Town on the Message Bus

    -

    In this chapter, we’ll start to make events more fundamental to the internal +

    + +In this chapter, we’ll start to make events more fundamental to the internal structure of our application. We’ll move from the current state in Before: the message bus is an optional add-on, where events are an optional side effect…​

    @@ -122,7 +226,9 @@

    Going to Town on the Message Bus

    Figure 1. Before: the message bus is an optional add-on
    -

    …​to the situation in The message bus is now the main entrypoint to the service layer, where +

    + +…​to the situation in The message bus is now the main entrypoint to the service layer, where everything goes via the message bus, and our app has been transformed fundamentally into a message processor.

    @@ -157,9 +263,11 @@

    Going to Town on the Message Bus

    -

    A New Requirement Leads Us to a New Architecture

    +

    A New Requirement Leads Us to a New Architecture

    -

    Rich Hickey talks about situated software, meaning software that runs for +

    + +Rich Hickey talks about situated software, meaning software that runs for extended periods of time, managing a real-world process. Examples include warehouse-management systems, logistics schedulers, and payroll systems.

    @@ -185,11 +293,13 @@

    A New Requirement Lea

    -

    In these types of situations, we learn about the need to change batch quantities +

    +In these types of situations, we learn about the need to change batch quantities when they’re already in the system. Perhaps someone made a mistake on the number in the manifest, or perhaps some sofas fell off a truck. Following a -conversation with the business,[1] we model the situation as in -Batch quantity changed means deallocate and reallocate.

    +conversation with the business,[1] + +we model the situation as in Batch quantity changed means deallocate and reallocate.

    @@ -227,9 +337,11 @@

    A New Requirement Lea transactions and data integrity.

    -

    Imagining an Architecture Change: Everything Will Be an Event Handler

    +

    Imagining an Architecture Change: Everything Will Be an Event Handler

    -

    But before we jump in, think about where we’re headed. There are two +

    + +But before we jump in, think about where we’re headed. There are two kinds of flows through our system:

    @@ -244,7 +356,8 @@

    I

    -

    Wouldn’t it be easier if everything was an event handler? If we rethink our API +

    +Wouldn’t it be easier if everything was an event handler? If we rethink our API calls as capturing events, the service-layer functions can be event handlers too, and we no longer need to make a distinction between internal and external event handlers:

    @@ -257,7 +370,8 @@

    I
  • services.add_batch() could be the handler for a BatchCreated -event.[2]

    +event.[2] +

  • @@ -268,21 +382,23 @@

    I
    • An event called BatchQuantityChanged can invoke a handler called -change_batch_quantity().

      +change_batch_quantity(). +

    • And the new AllocationRequired events that it may raise can be passed on to services.allocate() too, so there is no conceptual difference between a brand-new allocation coming from the API and a reallocation that’s -internally triggered by a deallocation.

      +internally triggered by a deallocation. +

    -

    All sound like a bit much? Let’s work toward it all gradually. We’ll -follow the -Preparatory -Refactoring workflow, aka "Make the change easy; then make the easy change":

    +

    +All sound like a bit much? Let’s work toward it all gradually. We’ll +follow the Preparatory Refactoring workflow, aka "Make +the change easy; then make the easy change":

      @@ -311,9 +427,14 @@

      I

    -

    Refactoring Service Functions to Message Handlers

    +

    Refactoring Service Functions to Message Handlers

    -

    We start by defining the two events that capture our current API inputs—AllocationRequired and BatchCreated:

    +

    + + + +We start by defining the two events that capture our current API +inputs—AllocationRequired and BatchCreated:

    BatchCreated and AllocationRequired events (src/allocation/domain/events.py)
    @@ -325,7 +446,7 @@

    Refactoring Service ref: str sku: str qty: int - eta: Optional[date] = None + eta: Optional[date] = None ... @@ -349,7 +470,8 @@

    Refactoring Service
    def add_batch(
    -        event: events.BatchCreated, uow: unit_of_work.AbstractUnitOfWork
    +    event: events.BatchCreated,
    +    uow: unit_of_work.AbstractUnitOfWork,
     ):
         with uow:
             product = uow.products.get(sku=event.sku)
    @@ -357,18 +479,20 @@ 

    Refactoring Service def allocate( - event: events.AllocationRequired, uow: unit_of_work.AbstractUnitOfWork + event: events.AllocationRequired, + uow: unit_of_work.AbstractUnitOfWork, ) -> str: line = OrderLine(event.orderid, event.sku, event.qty) ... def send_out_of_stock_notification( - event: events.OutOfStock, uow: unit_of_work.AbstractUnitOfWork, + event: events.OutOfStock, + uow: unit_of_work.AbstractUnitOfWork, ): email.send( - 'stock@made.com', - f'Out of stock for {event.sku}', + "stock@made.com", + f"Out of stock for {event.sku}", )

    @@ -382,32 +506,33 @@

    Refactoring Service
    -
     def add_batch(
    --        ref: str, sku: str, qty: int, eta: Optional[date],
    --        uow: unit_of_work.AbstractUnitOfWork
    -+        event: events.BatchCreated, uow: unit_of_work.AbstractUnitOfWork
    - ):
    -     with uow:
    +
     def add_batch(
    +-    ref: str, sku: str, qty: int, eta: Optional[date],
    ++    event: events.BatchCreated,
    +     uow: unit_of_work.AbstractUnitOfWork,
    + ):
    +     with uow:
     -        product = uow.products.get(sku=sku)
     +        product = uow.products.get(sku=event.sku)
    -     ...
    +     ...
     
     
    - def allocate(
    --        orderid: str, sku: str, qty: int,
    --        uow: unit_of_work.AbstractUnitOfWork
    -+        event: events.AllocationRequired, uow: unit_of_work.AbstractUnitOfWork
    - ) -> str:
    + def allocate(
    +-    orderid: str, sku: str, qty: int,
    ++    event: events.AllocationRequired,
    +     uow: unit_of_work.AbstractUnitOfWork,
    + ) -> str:
     -    line = OrderLine(orderid, sku, qty)
     +    line = OrderLine(event.orderid, event.sku, event.qty)
    -     ...
    +     ...
     
     +
     +def send_out_of_stock_notification(
    -+        event: events.OutOfStock, uow: unit_of_work.AbstractUnitOfWork,
    ++    event: events.OutOfStock,
    ++    uow: unit_of_work.AbstractUnitOfWork,
     +):
     +    email.send(
    -     ...
    + ...
    @@ -420,12 +545,15 @@

    Refactoring Service
    From Domain Objects, via Primitive Obsession, to Events as an Interface
    -

    Some of you may remember [primitive_obsession], in which we changed our service-layer API +

    + + +Some of you may remember [primitive_obsession], in which we changed our service-layer API from being in terms of domain objects to primitives. And now we’re moving back, but to different objects? What gives?

    -

    In OO circles, people talk about primitive obsession as an anti-pattern: avoid +

    In OO circles, people talk about primitive obsession as an antipattern: avoid primitives in public APIs, and instead wrap them with custom value classes, they would say. In the Python world, a lot of people would be quite skeptical of that as a rule of thumb. When mindlessly applied, it’s certainly a recipe for @@ -453,19 +581,27 @@

    Refactoring Service

    -

    The Message Bus Now Collects Events from the UoW

    +

    The Message Bus Now Collects Events from the UoW

    -

    Our event handlers now need a UoW. In addition, as our message bus becomes +

    + + +Our event handlers now need a UoW. In addition, as our message bus becomes more central to our application, it makes sense to put it explicitly in charge of collecting and processing new events. There was a bit of a circular dependency -between the UoW and message bus until now, so this will make it one-way:

    +between the UoW and message bus until now, so this will make it one-way. Instead +of having the UoW push events onto the message bus, we will have the message +bus pull events from the UoW.

    Handle takes a UoW and manages a queue (src/allocation/service_layer/messagebus.py)
    -
    def handle(event: events.Event, uow: unit_of_work.AbstractUnitOfWork):  #(1)
    +
    def handle(
    +    event: events.Event,
    +    uow: unit_of_work.AbstractUnitOfWork,  #(1)
    +):
         queue = [event]  #(2)
         while queue:
             event = queue.pop(0)  #(3)
    @@ -507,20 +643,19 @@ 

    The Message Bus Now C
    -from . import messagebus  #(1)
    --
     
     
    - class AbstractUnitOfWork(abc.ABC):
    -@@ -23,13 +21,11 @@ class AbstractUnitOfWork(abc.ABC):
    + class AbstractUnitOfWork(abc.ABC):
    +@@ -22,13 +21,11 @@ class AbstractUnitOfWork(abc.ABC):
     
    -     def commit(self):
    -         self._commit()
    +     def commit(self):
    +         self._commit()
     -        self.publish_events()  #(2)
     
     -    def publish_events(self):
     +    def collect_new_events(self):
    -         for product in self.products.seen:
    -             while product.events:
    +         for product in self.products.seen:
    +             while product.events:
     -                event = product.events.pop(0)
     -                messagebus.handle(event)
     +                yield product.events.pop(0)  #(3)
    @@ -545,9 +680,11 @@

    The Message Bus Now C

    -

    Our Tests Are All Written in Terms of Events Too

    +

    Our Tests Are All Written in Terms of Events Too

    -

    Our tests now operate by creating events and putting them on the +

    + +Our tests now operate by creating events and putting them on the message bus, rather than invoking service-layer functions directly:

    @@ -556,22 +693,20 @@

    Our Tests Are All Wri
    class TestAddBatch:
    -
    -     def test_for_new_product(self):
    -         uow = FakeUnitOfWork()
    +     def test_for_new_product(self):
    +         uow = FakeUnitOfWork()
     -        services.add_batch("b1", "CRUNCHY-ARMCHAIR", 100, None, uow)
     +        messagebus.handle(
     +            events.BatchCreated("b1", "CRUNCHY-ARMCHAIR", 100, None), uow
     +        )
    -         assert uow.products.get("CRUNCHY-ARMCHAIR") is not None
    -         assert uow.committed
    +         assert uow.products.get("CRUNCHY-ARMCHAIR") is not None
    +         assert uow.committed
     
     ...
     
    - class TestAllocate:
    -
    -     def test_returns_allocation(self):
    -         uow = FakeUnitOfWork()
    + class TestAllocate:
    +     def test_returns_allocation(self):
    +         uow = FakeUnitOfWork()
     -        services.add_batch("batch1", "COMPLICATED-LAMP", 100, None, uow)
     -        result = services.allocate("o1", "COMPLICATED-LAMP", 10, uow)
     +        messagebus.handle(
    @@ -580,16 +715,18 @@ 

    Our Tests Are All Wri + result = messagebus.handle( + events.AllocationRequired("o1", "COMPLICATED-LAMP", 10), uow + ) - assert result == "batch1"

    + assert result == "batch1"

    -

    A Temporary Ugly Hack: The Message Bus Has to Return Results

    +

    A Temporary Ugly Hack: The Message Bus Has to Return Results

    -

    Our API and our service layer currently want to know the allocated batch reference +

    + +Our API and our service layer currently want to know the allocated batch reference when they invoke our allocate() handler. This means we need to put in a temporary hack on our message bus to let it return events:

    @@ -598,46 +735,51 @@

    A Temporary Ugly Hack: The Message Bus Has to Retur
    -
     def handle(event: events.Event, uow: unit_of_work.AbstractUnitOfWork):
    +
     def handle(
    +     event: events.Event,
    +     uow: unit_of_work.AbstractUnitOfWork,
    + ):
     +    results = []
    -     queue = [event]
    -     while queue:
    -         event = queue.pop(0)
    -         for handler in HANDLERS[type(event)]:
    +     queue = [event]
    +     while queue:
    +         event = queue.pop(0)
    +         for handler in HANDLERS[type(event)]:
     -            handler(event, uow=uow)
     +            results.append(handler(event, uow=uow))
    -             queue.extend(uow.collect_new_events())
    +             queue.extend(uow.collect_new_events())
     +    return results

    -

    It’s because we’re mixing the read and write responsibilities in our system. +

    + +It’s because we’re mixing the read and write responsibilities in our system. We’ll come back to fix this wart in [chapter_12_cqrs].

    -

    Modifying Our API to Work with Events

    +

    Modifying Our API to Work with Events

    Flask changing to message bus as a diff (src/allocation/entrypoints/flask_app.py)
    -
     @app.route("/allocate", methods=['POST'])
    - def allocate_endpoint():
    -     try:
    +
     @app.route("/allocate", methods=["POST"])
    + def allocate_endpoint():
    +     try:
     -        batchref = services.allocate(
    --            request.json['orderid'],  #(1)
    --            request.json['sku'],
    --            request.json['qty'],
    +-            request.json["orderid"],  #(1)
    +-            request.json["sku"],
    +-            request.json["qty"],
     -            unit_of_work.SqlAlchemyUnitOfWork(),
     +        event = events.AllocationRequired(  #(2)
    -+            request.json['orderid'], request.json['sku'], request.json['qty'],
    -         )
    ++            request.json["orderid"], request.json["sku"], request.json["qty"]
    +         )
     +        results = messagebus.handle(event, unit_of_work.SqlAlchemyUnitOfWork())  #(3)
     +        batchref = results.pop(0)
    -     except InvalidSku as e:
    + except InvalidSku as e:
    @@ -683,9 +825,11 @@

    Modifying Our API to Work with E

    -

    Implementing Our New Requirement

    +

    Implementing Our New Requirement

    -

    We’re done with our refactoring phase. Let’s see if we really have "made the +

    + +We’re done with our refactoring phase. Let’s see if we really have "made the change easy." Let’s implement our new requirement, shown in Sequence diagram for reallocation flow: we’ll receive as our inputs some new BatchQuantityChanged events and pass them to a handler, which in turn might emit some AllocationRequired events, and those in turn will go @@ -731,14 +875,17 @@

    Implementing Our New Requirement

    but the second one does not. You’ll need to think about whether this is acceptable, and whether you need to notice when it happens and do something about it. See [footguns] for more discussion. + +
    -

    Our New Event

    +

    Our New Event

    -

    The event that tells us a batch quantity has changed is simple; it just +

    +The event that tells us a batch quantity has changed is simple; it just needs a batch reference and a new quantity:

    @@ -757,9 +904,13 @@

    Our New Event

    -

    Test-Driving a New Handler

    +

    Test-Driving a New Handler

    -

    Following the lessons learned in [chapter_04_service_layer], +

    + + + +Following the lessons learned in [chapter_04_service_layer], we can operate in "high gear" and write our unit tests at the highest possible level of abstraction, in terms of events. Here’s what they might look like:

    @@ -770,11 +921,10 @@

    Test-Driving a New Handler

    class TestChangeBatchQuantity:
    -
         def test_changes_available_quantity(self):
             uow = FakeUnitOfWork()
             messagebus.handle(
    -            events.BatchCreated("batch1", "ADORABLE-SETTEE", 100, None), uow
    +            events.BatchCreated("batch1", "ADORABLE-SETTEE", 100, None), uow
             )
             [batch] = uow.products.get(sku="ADORABLE-SETTEE").batches
             assert batch.available_quantity == 100  #(1)
    @@ -783,11 +933,10 @@ 

    Test-Driving a New Handler

    assert batch.available_quantity == 50 #(1) - def test_reallocates_if_necessary(self): uow = FakeUnitOfWork() event_history = [ - events.BatchCreated("batch1", "INDIFFERENT-TABLE", 50, None), + events.BatchCreated("batch1", "INDIFFERENT-TABLE", 50, None), events.BatchCreated("batch2", "INDIFFERENT-TABLE", 50, date.today()), events.AllocationRequired("order1", "INDIFFERENT-TABLE", 20), events.AllocationRequired("order2", "INDIFFERENT-TABLE", 20), @@ -822,9 +971,10 @@

    Test-Driving a New Handler

    -

    Implementation

    +

    Implementation

    -

    Our new handler is very simple:

    +

    +Our new handler is very simple:

    Handler delegates to model layer (src/allocation/service_layer/handlers.py)
    @@ -832,7 +982,8 @@

    Implementation

    def change_batch_quantity(
    -        event: events.BatchQuantityChanged, uow: unit_of_work.AbstractUnitOfWork
    +    event: events.BatchQuantityChanged,
    +    uow: unit_of_work.AbstractUnitOfWork,
     ):
         with uow:
             product = uow.products.get_by_batchref(batchref=event.ref)
    @@ -843,7 +994,8 @@ 

    Implementation

    -

    We realize we’ll need a new query type on our repository:

    +

    +We realize we’ll need a new query type on our repository:

    A new query type on our repository (src/allocation/adapters/repository.py)
    @@ -856,23 +1008,23 @@

    Implementation

    def get(self, sku) -> model.Product: ... - def get_by_batchref(self, batchref) -> model.Product: - product = self._get_by_batchref(batchref) + def get_by_batchref(self, batchref) -> model.Product: + product = self._get_by_batchref(batchref) if product: self.seen.add(product) return product - @abc.abstractmethod + @abc.abstractmethod def _add(self, product: model.Product): raise NotImplementedError - @abc.abstractmethod + @abc.abstractmethod def _get(self, sku) -> model.Product: raise NotImplementedError - @abc.abstractmethod - def _get_by_batchref(self, batchref) -> model.Product: - raise NotImplementedError + @abc.abstractmethod + def _get_by_batchref(self, batchref) -> model.Product: + raise NotImplementedError ... class SqlAlchemyRepository(AbstractRepository): @@ -881,16 +1033,20 @@

    Implementation

    def _get(self, sku): return self.session.query(model.Product).filter_by(sku=sku).first() - def _get_by_batchref(self, batchref): - return self.session.query(model.Product).join(model.Batch).filter( - orm.batches.c.reference == batchref, - ).first()
    + def _get_by_batchref(self, batchref): + return ( + self.session.query(model.Product) + .join(model.Batch) + .filter(orm.batches.c.reference == batchref) + .first() + )
    -

    And on our FakeRepository too:

    +

    +And on our FakeRepository too:

    Updating the fake repo too (tests/unit/test_handlers.py)
    @@ -901,13 +1057,13 @@

    Implementation

    ... def _get(self, sku): - return next((p for p in self._products if p.sku == sku), None) + return next((p for p in self._products if p.sku == sku), None) def _get_by_batchref(self, batchref): - return next(( - p for p in self._products for b in p.batches - if b.reference == batchref - ), None)
    + return next( + (p for p in self._products for b in p.batches if b.reference == batchref), + None, + )
    @@ -920,21 +1076,27 @@

    Implementation

    We’re adding a query to our repository to make this use case easier to -implement. So long as our query is returning a single aggregate, we’re not -bending any rules. If you find yourself writing complex queries on your -repositories, you might want to consider a different design. Methods like get_most_popular_products or find_products_by_order_id in particular would -definitely trigger our spidey sense. [chapter_11_external_events] and the epilogue have some tips on managing complex queries. + implement. So long as our query is returning a single aggregate, we’re not + bending any rules. If you find yourself writing complex queries on your + repositories, you might want to consider a different design. Methods like + get_most_popular_products or find_products_by_order_id in particular + would definitely trigger our spidey sense. [chapter_11_external_events] + and the epilogue have some tips + on managing complex queries. +
    -

    A New Method on the Domain Model

    +

    A New Method on the Domain Model

    -

    We add the new method to the model, which does the quantity change and -deallocation(s) inline and publishes a new event. We also modify the existing -allocate function to publish an event:

    +

    +We add the new method to the model, +which does the quantity change and deallocation(s) inline +and publishes a new event. +We also modify the existing allocate function to publish an event:

    Our model evolves to capture the new requirement (src/allocation/domain/model.py)
    @@ -964,7 +1126,8 @@

    A New Method on the Domain Model

    -

    We wire up our new handler:

    +

    +We wire up our new handler:

    The message bus grows (src/allocation/service_layer/messagebus.py)
    @@ -976,7 +1139,6 @@

    A New Method on the Domain Model

    events.BatchQuantityChanged: [handlers.change_batch_quantity], events.AllocationRequired: [handlers.allocate], events.OutOfStock: [handlers.send_out_of_stock_notification], - } # type: Dict[Type[events.Event], List[Callable]]
    @@ -988,9 +1150,12 @@

    A New Method on the Domain Model

    -

    Optionally: Unit Testing Event Handlers in Isolation with a Fake Message Bus

    +

    Optionally: Unit Testing Event Handlers in Isolation with a Fake Message Bus

    -

    Our main test for the reallocation workflow is edge-to-edge +

    + + +Our main test for the reallocation workflow is edge-to-edge (see the example code in Test-Driving a New Handler). It uses the real message bus, and it tests the whole flow, where the BatchQuantityChanged event handler triggers deallocation, and emits new AllocationRequired events, which in @@ -1003,7 +1168,8 @@

    Optionally: Unit Testing Event Handlers in Isolation w using a "fake" message bus.

    -

    In our case, we actually intervene by modifying the publish_events() method +

    +In our case, we actually intervene by modifying the publish_events() method on FakeUnitOfWork and decoupling it from the real message bus, instead making it record what events it sees:

    @@ -1013,7 +1179,6 @@

    Optionally: Unit Testing Event Handlers in Isolation w
    class FakeUnitOfWorkWithFakeMessageBus(FakeUnitOfWork):
    -
         def __init__(self):
             super().__init__()
             self.events_published = []  # type: List[events.Event]
    @@ -1027,7 +1192,8 @@ 

    Optionally: Unit Testing Event Handlers in Isolation w

    -

    Now when we invoke messagebus.handle() using the FakeUnitOfWorkWithFakeMessageBus, +

    +Now when we invoke messagebus.handle() using the FakeUnitOfWorkWithFakeMessageBus, it runs only the handler for that event. So we can write a more isolated unit test: instead of checking all the side effects, we just check that BatchQuantityChanged leads to AllocationRequired if the quantity drops @@ -1043,7 +1209,7 @@

    Optionally: Unit Testing Event Handlers in Isolation w # test setup as before event_history = [ - events.BatchCreated("batch1", "INDIFFERENT-TABLE", 50, None), + events.BatchCreated("batch1", "INDIFFERENT-TABLE", 50, None), events.BatchCreated("batch2", "INDIFFERENT-TABLE", 50, date.today()), events.AllocationRequired("order1", "INDIFFERENT-TABLE", 20), events.AllocationRequired("order2", "INDIFFERENT-TABLE", 20), @@ -1059,8 +1225,8 @@

    Optionally: Unit Testing Event Handlers in Isolation w # assert on new events emitted rather than downstream side-effects [reallocation_event] = uow.events_published assert isinstance(reallocation_event, events.AllocationRequired) - assert reallocation_event.orderid in {'order1', 'order2'} - assert reallocation_event.sku == 'INDIFFERENT-TABLE' + assert reallocation_event.orderid in {"order1", "order2"} + assert reallocation_event.sku == "INDIFFERENT-TABLE"

    @@ -1074,13 +1240,15 @@

    Optionally: Unit Testing Event Handlers in Isolation w
    Exercise for the Reader
    -

    A great way to force yourself to really understand some code is to refactor it. +

    +A great way to force yourself to really understand some code is to refactor it. In the discussion of testing handlers in isolation, we used something called FakeUnitOfWorkWithFakeMessageBus, which is unnecessarily complicated and violates the SRP.

    -

    If we change the message bus to being a class,[3] +

    +If we change the message bus to being a class,[3] then building a FakeMessageBus is more straightforward:

    @@ -1106,7 +1274,7 @@

    Optionally: Unit Testing Event Handlers in Isolation w class FakeMessageBus(messagebus.AbstractMessageBus): def __init__(self): self.events_published = [] # type: List[events.Event] - self.handlers = { + self.HANDLERS = { events.OutOfStock: [lambda e: self.events_published.append(e)] }

    @@ -1127,12 +1295,12 @@

    Optionally: Unit Testing Event Handlers in Isolation w

    -

    Wrap-Up

    +

    Wrap-Up

    Let’s look back at what we’ve achieved, and think about why we did it.

    -

    What Have We Achieved?

    +

    What Have We Achieved?

    Events are simple dataclasses that define the data structures for inputs and internal messages within our system. This is quite powerful from a DDD @@ -1148,9 +1316,11 @@

    What Have We Achieved?

    -

    Why Have We Achieved?

    +

    Why Have We Achieved?

    -

    Our ongoing objective with these architectural patterns is to try to have +

    + +Our ongoing objective with these architectural patterns is to try to have the complexity of our application grow more slowly than its size. When we go all in on the message bus, as always we pay a price in terms of architectural complexity (see Whole app is a message bus: the trade-offs), but we buy ourselves a @@ -1207,7 +1377,8 @@

    Why Have We Achieved?

    -

    Now, you may be wondering, where are those BatchQuantityChanged events +

    +Now, you may be wondering, where are those BatchQuantityChanged events going to come from? The answer is revealed in a couple chapters' time. But first, let’s talk about events versus commands.

    @@ -1215,6 +1386,11 @@

    Why Have We Achieved?

    +

    @@ -1222,7 +1398,7 @@

    Why Have We Achieved?

    1. Event-based modeling is so popular that a practice called event storming has been developed for facilitating event-based requirements gathering and domain model elaboration.
    -2. If you’ve done a bit of reading about event-driven architectures, you may be thinking, "Some of these events sound more like commands!" Bear with us! We’re trying to introduce one concept at a time. In the next chapter, we’ll introduce the distinction between commands and events. +2. If you’ve done a bit of reading about event-driven architectures, you may be thinking, "Some of these events sound more like commands!" Bear with us! We’re trying to introduce one concept at a time. In the next chapter, we’ll introduce the distinction between commands and events.
    3. The "simple" implementation in this chapter essentially uses the messagebus.py module itself to implement the Singleton Pattern. @@ -1230,93 +1406,22 @@

    Why Have We Achieved?

    -
    diff --git a/docs/book/chapter_10_commands.html b/book/chapter_10_commands.html similarity index 88% rename from docs/book/chapter_10_commands.html rename to book/chapter_10_commands.html index 5279b35..cab9dcb 100644 --- a/docs/book/chapter_10_commands.html +++ b/book/chapter_10_commands.html @@ -3,7 +3,7 @@ - + Commands and Command Handler + @@ -81,7 +183,7 @@
    -

    Commands and Command Handler

    +

    10: Commands and Command Handler

    -

    In the previous chapter, we talked about using events as a way of representing +

    +In the previous chapter, we talked about using events as a way of representing the inputs to our system, and we turned our application into a message-processing machine.

    @@ -148,9 +251,11 @@

    Commands and Command Handler

    -

    Commands and Events

    +

    Commands and Events

    -

    Like events, commands are a type of message—​instructions sent by one part of +

    + +Like events, commands are a type of message—​instructions sent by one part of a system to another. We usually represent commands with dumb data structures and can handle them in much the same way as events.

    @@ -213,7 +318,9 @@

    Commands and Events

    -

    What kinds of commands do we have in our system right now?

    +

    + +What kinds of commands do we have in our system right now?

    Pulling out some commands (src/allocation/domain/commands.py)
    @@ -223,18 +330,21 @@

    Commands and Events

    class Command:
         pass
     
    +
     @dataclass
     class Allocate(Command):  #(1)
         orderid: str
         sku: str
         qty: int
     
    +
     @dataclass
     class CreateBatch(Command):  #(2)
         ref: str
         sku: str
         qty: int
    -    eta: Optional[date] = None
    +    eta: Optional[date] = None
    +
     
     @dataclass
     class ChangeBatchQuantity(Command):  #(3)
    @@ -259,9 +369,12 @@ 

    Commands and Events

    -

    Differences in Exception Handling

    +

    Differences in Exception Handling

    -

    Just changing the names and verbs is all very well, but that won’t +

    + + +Just changing the names and verbs is all very well, but that won’t change the behavior of our system. We want to treat events and commands similarly, but not exactly the same. Let’s see how our message bus changes:

    @@ -273,7 +386,10 @@

    Differences in Exception HandlingMessage = Union[commands.Command, events.Event] -def handle(message: Message, uow: unit_of_work.AbstractUnitOfWork): #(1) +def handle( #(1) + message: Message, + uow: unit_of_work.AbstractUnitOfWork, +): results = [] queue = [message] while queue: @@ -284,7 +400,7 @@

    Differences in Exception Handlingcmd_result = handle_command(message, queue, uow) #(2) results.append(cmd_result) else: - raise Exception(f'{message} was not an Event or Command') + raise Exception(f"{message} was not an Event or Command") return results

    @@ -312,15 +428,15 @@

    Differences in Exception Handlingdef handle_event( event: events.Event, queue: List[Message], - uow: unit_of_work.AbstractUnitOfWork + uow: unit_of_work.AbstractUnitOfWork, ): for handler in EVENT_HANDLERS[type(event)]: #(1) try: - logger.debug('handling event %s with handler %s', event, handler) + logger.debug("handling event %s with handler %s", event, handler) handler(event, uow=uow) queue.extend(uow.collect_new_events()) except Exception: - logger.exception('Exception handling event %s', event) + logger.exception("Exception handling event %s", event) continue #(2)

    @@ -339,7 +455,8 @@

    Differences in Exception Handling
    -

    And here’s how we do commands:

    +

    +And here’s how we do commands:

    Commands reraise exceptions (src/allocation/service_layer/messagebus.py)
    @@ -349,16 +466,16 @@

    Differences in Exception Handlingdef handle_command( command: commands.Command, queue: List[Message], - uow: unit_of_work.AbstractUnitOfWork + uow: unit_of_work.AbstractUnitOfWork, ): - logger.debug('handling command %s', command) + logger.debug("handling command %s", command) try: handler = COMMAND_HANDLERS[type(command)] #(1) result = handler(command, uow=uow) queue.extend(uow.collect_new_events()) return result #(3) except Exception: - logger.exception('Exception handling command %s', command) + logger.exception("Exception handling command %s", command) raise #(2)

    @@ -380,7 +497,10 @@

    Differences in Exception Handling
    -

    We also change the single HANDLERS dict into different ones for +

    + + +We also change the single HANDLERS dict into different ones for commands and events. Commands can have only one handler, according to our convention:

    @@ -404,9 +524,12 @@

    Differences in Exception Handling
    -

    Discussion: Events, Commands, and Error Handling

    +

    Discussion: Events, Commands, and Error Handling

    -

    Many developers get uncomfortable at this point and ask, "What happens when an +

    + + +Many developers get uncomfortable at this point and ask, "What happens when an event fails to process? How am I supposed to make sure the system is in a consistent state?" If we manage to process half of the events during messagebus.handle before an out-of-memory error kills our process, how do we mitigate problems caused by the @@ -426,13 +549,17 @@

    Discussion: Events, Comm same beanbags to another customer, causing a headache for customer support.

    -

    In our allocation service, though, we’ve already taken steps to prevent that +

    + + +In our allocation service, though, we’ve already taken steps to prevent that happening. We’ve carefully identified aggregates that act as consistency boundaries, and we’ve introduced a UoW that manages the atomic success or failure of an update to an aggregate.

    -

    For example, when we allocate stock to an order, our consistency boundary is the +

    +For example, when we allocate stock to an order, our consistency boundary is the Product aggregate. This means that we can’t accidentally overallocate: either a particular order line is allocated to the product, or it is not—​there’s no room for inconsistent states.

    @@ -452,7 +579,7 @@

    Discussion: Events, Comm successful.

    -

    Let’s look at another example (from a different, imaginary projet) to see why not.

    +

    Let’s look at another example (from a different, imaginary project) to see why not.

    Imagine we are building an ecommerce website that sells expensive luxury goods. @@ -472,7 +599,8 @@

    Discussion: Events, Comm

    -

    Using the techniques we’ve already discussed in this book, we decide that we +

    +Using the techniques we’ve already discussed in this book, we decide that we want to build a new History aggregate that records orders and can raise domain events when rules are met. We will structure the code like this:

    @@ -484,7 +612,7 @@

    Discussion: Events, Comm
    class History:  # Aggregate
     
         def __init__(self, customer_id: int):
    -        self.orders = set() # Set[HistoryEntry]
    +        self.orders = set()  # Set[HistoryEntry]
             self.customer_id = customer_id
     
         def record_order(self, order_id: str, order_amount: int): #(1)
    @@ -505,14 +633,14 @@ 

    Discussion: Events, Comm with uow: order = Order.from_basket(cmd.customer_id, cmd.basket_items) uow.orders.add(order) - uow.commit() # raises OrderCreated + uow.commit() # raises OrderCreated def update_customer_history(uow, event: OrderCreated): #(3) with uow: history = uow.order_history.get(event.customer_id) history.record_order(event.order_id, event.order_amount) - uow.commit() # raises CustomerBecameVIP + uow.commit() # raises CustomerBecameVIP def congratulate_vip_customer(uow, event: CustomerBecameVip): #(4) @@ -520,7 +648,7 @@

    Discussion: Events, Comm customer = uow.customers.get(event.customer_id) email.send( customer.email_address, - f'Congratulations {customer.first_name}!' + f'Congratulations {customer.first_name}!' )

    @@ -551,7 +679,8 @@

    Discussion: Events, Comm event-driven system.

    -

    In our current implementation, we raise events about an aggregate after we +

    +In our current implementation, we raise events about an aggregate after we persist our state to the database. What if we raised those events before we persisted, and committed all our changes at the same time? That way, we could be sure that all the work was complete. Wouldn’t that be safer?

    @@ -573,7 +702,10 @@

    Discussion: Events, Comm our business stakeholders should prioritize.

    -

    Notice how we’ve deliberately aligned our transactional boundaries to the start +

    + + +Notice how we’ve deliberately aligned our transactional boundaries to the start and end of the business processes. The names that we use in the code match the jargon used by our business stakeholders, and the handlers we’ve written match the steps of our natural language acceptance criteria. This concordance of names @@ -582,9 +714,11 @@

    Discussion: Events, Comm

    -

    Recovering from Errors Synchronously

    +

    Recovering from Errors Synchronously

    -

    Hopefully we’ve convinced you that it’s OK for events to fail independently +

    + +Hopefully we’ve convinced you that it’s OK for events to fail independently from the commands that raised them. What should we do, then, to make sure we can recover from errors when they inevitably occur?

    @@ -593,7 +727,8 @@

    Recovering from Errors Synchronously

    usually rely on logs.

    -

    Let’s look again at the handle_event method from our message bus:

    +

    +Let’s look again at the handle_event method from our message bus:

    Current handle function (src/allocation/service_layer/messagebus.py)
    @@ -603,15 +738,15 @@

    Recovering from Errors Synchronously

    def handle_event(
         event: events.Event,
         queue: List[Message],
    -    uow: unit_of_work.AbstractUnitOfWork
    +    uow: unit_of_work.AbstractUnitOfWork,
     ):
         for handler in EVENT_HANDLERS[type(event)]:
             try:
    -            logger.debug('handling event %s with handler %s', event, handler)
    +            logger.debug("handling event %s with handler %s", event, handler)
                 handler(event, uow=uow)
                 queue.extend(uow.collect_new_events())
             except Exception:
    -            logger.exception('Exception handling event %s', event)
    +            logger.exception("Exception handling event %s", event)
                 continue
    @@ -629,7 +764,8 @@

    Recovering from Errors Synchronously

    -

    Because we’ve chosen to use dataclasses for our message types, we get a neatly +

    +Because we’ve chosen to use dataclasses for our message types, we get a neatly printed summary of the incoming data that we can copy and paste into a Python shell to re-create the object.

    @@ -644,7 +780,9 @@

    Recovering from Errors Synchronously

    deadlocks, and brief downtime caused by deployments.

    -

    For most of those cases, we can recover elegantly by trying again. As the +

    + +For most of those cases, we can recover elegantly by trying again. As the proverb says, "If at first you don’t succeed, retry the operation with an exponentially increasing back-off period."

    @@ -660,9 +798,8 @@

    Recovering from Errors Synchronously

    def handle_event( event: events.Event, queue: List[Message], - uow: unit_of_work.AbstractUnitOfWork + uow: unit_of_work.AbstractUnitOfWork, ): - for handler in EVENT_HANDLERS[type(event)]: try: for attempt in Retrying( #(2) @@ -671,12 +808,12 @@

    Recovering from Errors Synchronously

    ): with attempt: - logger.debug('handling event %s with handler %s', event, handler) + logger.debug("handling event %s with handler %s", event, handler) handler(event, uow=uow) queue.extend(uow.collect_new_events()) except RetryError as retry_failure: logger.error( - 'Failed to handle event %s times, giving up!, + "Failed to handle event %s times, giving up!", retry_failure.last_attempt.attempt_number ) continue
    @@ -687,7 +824,9 @@

    Recovering from Errors Synchronously

    1. -

      Tenacity is a Python library that implements common patterns for retrying.

      +

      Tenacity is a Python library that implements common patterns for retrying. + +

    2. Here we configure our message bus to retry operations up to three times, @@ -718,9 +857,12 @@

      Recovering from Errors Synchronously

    -

    Wrap-Up

    +

    Wrap-Up

    -

    In this book we decided to introduce the concept of events before the concept +

    + + +In this book we decided to introduce the concept of events before the concept of commands, but other guides often do it the other way around. Making explicit the requests that our system can respond to by giving them a name and their own data structure is quite a fundamental thing to do. You’ll @@ -767,7 +909,8 @@

    Wrap-Up

  • We’re expressly inviting failure. We know that sometimes things will break, and we’re choosing to handle that by making the failures smaller and more isolated. -This can make the system harder to reason about and requires better monitoring.

    +This can make the system harder to reason about and requires better monitoring. +

  • @@ -780,96 +923,30 @@

    Wrap-Up

    + -
    diff --git a/docs/book/chapter_11_external_events.html b/book/chapter_11_external_events.html similarity index 83% rename from docs/book/chapter_11_external_events.html rename to book/chapter_11_external_events.html index d49ebfa..9ee2654 100644 --- a/docs/book/chapter_11_external_events.html +++ b/book/chapter_11_external_events.html @@ -3,7 +3,7 @@ - + Event-Driven Architecture: Using Events to Integrate Microservices + @@ -81,7 +183,7 @@
    -

    Event-Driven Architecture: Using Events to Integrate Microservices

    +

    11: Event-Driven Architecture: Using Events to Integrate Microservices

    -

    In the preceding chapter, we never actually spoke about how we would receive +

    + + +In the preceding chapter, we never actually spoke about how we would receive the "batch quantity changed" events, or indeed, how we might notify the outside world about reallocations.

    @@ -161,9 +266,13 @@

    Event-Driven Architecture: Using Events to I

    -

    Distributed Ball of Mud, and Thinking in Nouns

    +

    Distributed Ball of Mud, and Thinking in Nouns

    -

    Before we get into that, let’s talk about the alternatives. We regularly talk to +

    + + + +Before we get into that, let’s talk about the alternatives. We regularly talk to engineers who are trying to build out a microservices architecture. Often they are migrating from an existing application, and their first instinct is to split their system into nouns.

    @@ -202,7 +311,8 @@

    Distributed Ball of Mud,

    Each "thing" in our system has an associated service, which exposes an HTTP API.

    -

    Let’s work through an example happy-path flow in Command flow 1: +

    +Let’s work through an example happy-path flow in Command flow 1: our users visit a website and can choose from products that are in stock. When they add an item to their basket, we will reserve some stock for them. When an order is complete, we confirm the reservation, which causes us to send dispatch @@ -270,7 +380,8 @@

    Distributed Ball of Mud,

    Where does this logic go?

    -

    Well, the Warehouse system knows that the stock has been damaged, so maybe it +

    +Well, the Warehouse system knows that the stock has been damaged, so maybe it should own this process, as shown in Command flow 2.

    @@ -311,13 +422,19 @@

    Distributed Ball of Mud,

    Multiply this by all the other workflows we need to provide, and you can see -how services quickly get tangled up.

    +how services quickly get tangled up. + + + +

    -

    Error Handling in Distributed Systems

    +

    Error Handling in Distributed Systems

    -

    "Things break" is a universal law of software engineering. What happens in our +

    + +"Things break" is a universal law of software engineering. What happens in our system when one of our requests fails? Let’s say that a network error happens right after we take a user’s order for three MISBEGOTTEN-RUG, as shown in Command flow with error.

    @@ -329,7 +446,10 @@

    Error Handling in Distributed Sy affecting the reliability of our order service.

    -

    When two things have to be changed together, we say that they are coupled. We +

    + + +When two things have to be changed together, we say that they are coupled. We can think of this failure cascade as a kind of temporal coupling: every part of the system has to work at the same time for any part of it to work. As the system gets bigger, there is an exponentially increasing probability that some @@ -363,7 +483,8 @@

    Error Handling in Distributed Sy
    Connascence
    -

    We’re using the term coupling here, but there’s another way to describe +

    +We’re using the term coupling here, but there’s another way to describe the relationships between our systems. Connascence is a term used by some authors to describe the different types of coupling.

    @@ -389,7 +510,8 @@

    Error Handling in Distributed Sy it carries.

    -

    We can never completely avoid coupling, except by having our software not talk +

    +We can never completely avoid coupling, except by having our software not talk to any other software. What we want is to avoid inappropriate coupling. Connascence provides a mental model for understanding the strength and type of coupling inherent in different architectural styles. Read all about it at @@ -399,9 +521,16 @@

    Error Handling in Distributed Sy

    -

    The Alternative: Temporal Decoupling Using Asynchronous Messaging

    +

    The Alternative: Temporal Decoupling Using Asynchronous Messaging

    -

    How do we get appropriate coupling? We’ve already seen part of the answer, which is that we should think in +

    + + + + + + +How do we get appropriate coupling? We’ve already seen part of the answer, which is that we should think in terms of verbs, not nouns. Our domain model is about modeling a business process. It’s not a static data model about a thing; it’s a model of a verb.

    @@ -430,14 +559,16 @@

    The A

    -

    Like aggregates, microservices should be consistency boundaries. Between two +

    + +Like aggregates, microservices should be consistency boundaries. Between two services, we can accept eventual consistency, and that means we don’t need to rely on synchronous calls. Each service accepts commands from the outside world and raises events to record the result. Other services can listen to those events to trigger the next steps in the workflow.

    -

    To avoid the Distributed Ball of Mud anti-pattern, instead of temporally coupled HTTP +

    To avoid the Distributed Ball of Mud antipattern, instead of temporally coupled HTTP API calls, we want to use asynchronous messaging to integrate our systems. We want our BatchQuantityChanged messages to come in as external messages from upstream systems, and we want our system to publish Allocated events for @@ -455,9 +586,14 @@

    The A

    -

    Using a Redis Pub/Sub Channel for Integration

    +

    Using a Redis Pub/Sub Channel for Integration

    -

    Let’s see how it will all work concretely. We’ll need some way of getting +

    + + + + +Let’s see how it will all work concretely. We’ll need some way of getting events out of one system and into another, like our message bus, but for services. This piece of infrastructure is often called a message broker. The role of a message broker is to take messages from publishers and deliver them @@ -521,9 +657,12 @@

    Using a Redis Pub/Sub Cha

    -

    Test-Driving It All Using an End-to-End Test

    +

    Test-Driving It All Using an End-to-End Test

    -

    Here’s how we might start with an end-to-end test. We can use our existing +

    + + +Here’s how we might start with an end-to-end test. We can use our existing API to create batches, and then we’ll test both inbound and outbound messages:

    @@ -534,30 +673,31 @@

    Test-Driving It All Using
    def test_change_batch_quantity_leading_to_reallocation():
         # start with two batches and an order allocated to one of them  #(1)
         orderid, sku = random_orderid(), random_sku()
    -    earlier_batch, later_batch = random_batchref('old'), random_batchref('newer')
    -    api_client.post_to_add_batch(earlier_batch, sku, qty=10, eta='2011-01-02')  #(2)
    -    api_client.post_to_add_batch(later_batch, sku, qty=10, eta='2011-01-02')
    +    earlier_batch, later_batch = random_batchref("old"), random_batchref("newer")
    +    api_client.post_to_add_batch(earlier_batch, sku, qty=10, eta="2011-01-01")  #(2)
    +    api_client.post_to_add_batch(later_batch, sku, qty=10, eta="2011-01-02")
         response = api_client.post_to_allocate(orderid, sku, 10)  #(2)
    -    assert response.json()['batchref'] == earlier_batch
    +    assert response.json()["batchref"] == earlier_batch
     
    -    subscription = redis_client.subscribe_to('line_allocated')  #(3)
    +    subscription = redis_client.subscribe_to("line_allocated")  #(3)
     
         # change quantity on allocated batch so it's less than our order  #(1)
    -    redis_client.publish_message('change_batch_quantity', {  #(3)
    -        'batchref': earlier_batch, 'qty': 5
    -    })
    +    redis_client.publish_message(  #(3)
    +        "change_batch_quantity",
    +        {"batchref": earlier_batch, "qty": 5},
    +    )
     
         # wait until we see a message saying the order has been reallocated  #(1)
         messages = []
    -    for attempt in Retrying(stop=stop_after_delay(3), reraise=True):  #(4)
    +    for attempt in Retrying(stop=stop_after_delay(3), reraise=True):  #(4)
             with attempt:
                 message = subscription.get_message(timeout=1)
                 if message:
                     messages.append(message)
    -                print(messages)
    -            data = json.loads(messages[-1]['data'])
    -            assert data['orderid'] == orderid
    -            assert data['batchref'] == later_batch
    + print(messages) + data = json.loads(messages[-1]["data"]) + assert data["orderid"] == orderid + assert data["batchref"] == later_batch

    @@ -590,9 +730,11 @@

    Test-Driving It All Using

    -

    Redis Is Another Thin Adapter Around Our Message Bus

    +

    Redis Is Another Thin Adapter Around Our Message Bus

    -

    Our Redis pub/sub listener (we call it an event consumer) is very much like +

    + +Our Redis pub/sub listener (we call it an event consumer) is very much like Flask: it translates from the outside world to our events:

    @@ -605,17 +747,17 @@

    Redis Is Another def main(): orm.start_mappers() - pubsub = r.pubsub(ignore_subscribe_messages=True) - pubsub.subscribe('change_batch_quantity') #(1) + pubsub = r.pubsub(ignore_subscribe_messages=True) + pubsub.subscribe("change_batch_quantity") #(1) for m in pubsub.listen(): handle_change_batch_quantity(m) def handle_change_batch_quantity(m): - logging.debug('handling %s', m) - data = json.loads(m['data']) #(2) - cmd = commands.ChangeBatchQuantity(ref=data['batchref'], qty=data['qty']) #(2) + logging.debug("handling %s", m) + data = json.loads(m["data"]) #(2) + cmd = commands.ChangeBatchQuantity(ref=data["batchref"], qty=data["qty"]) #(2) messagebus.handle(cmd, uow=unit_of_work.SqlAlchemyUnitOfWork())

    @@ -646,7 +788,7 @@

    Redis Is Another def publish(channel, event: events.Event): #(1) - logging.debug('publishing: channel=%s, event=%s', channel, event) + logging.debug("publishing: channel=%s, event=%s", channel, event) r.publish(channel, json.dumps(asdict(event))) @@ -663,9 +805,10 @@

    Redis Is Another
    -

    Our New Outgoing Event

    +

    Our New Outgoing Event

    -

    Here’s what the Allocated event will look like:

    +

    +Here’s what the Allocated event will look like:

    New event (src/allocation/domain/events.py)
    @@ -702,17 +845,22 @@

    Our New Outgoing Event

    batch.allocate(line) self.version_number += 1 - self.events.append(events.Allocated( - orderid=line.orderid, sku=line.sku, qty=line.qty, - batchref=batch.reference, - )) + self.events.append( + events.Allocated( + orderid=line.orderid, + sku=line.sku, + qty=line.qty, + batchref=batch.reference, + ) + ) return batch.reference
    -

    The handler for ChangeBatchQuantity already exists, so all we need to add +

    +The handler for ChangeBatchQuantity already exists, so all we need to add is a handler that publishes the outgoing event:

    @@ -729,7 +877,8 @@

    Our New Outgoing Event

    -

    Publishing the event uses our helper function from the Redis wrapper:

    +

    +Publishing the event uses our helper function from the Redis wrapper:

    Publish to Redis (src/allocation/service_layer/handlers.py)
    @@ -737,9 +886,10 @@

    Our New Outgoing Event

    def publish_allocated_event(
    -        event: events.Allocated, uow: unit_of_work.AbstractUnitOfWork,
    +    event: events.Allocated,
    +    uow: unit_of_work.AbstractUnitOfWork,
     ):
    -    redis_eventpublisher.publish('line_allocated', event)
    + redis_eventpublisher.publish("line_allocated", event)
    @@ -747,9 +897,11 @@

    Our New Outgoing Event

    -

    Internal Versus External Events

    +

    Internal Versus External Events

    -

    It’s a good idea to keep the distinction between internal and external events +

    + +It’s a good idea to keep the distinction between internal and external events clear. Some events may come from the outside, and some events may get upgraded and published externally, but not all of them will. This is particularly important if you get into @@ -779,19 +931,20 @@

    Internal Versus External Events

    You will likely want to add a new E2E test and feed through some changes into -redis_eventconsumer.py.

    +redis_eventconsumer.py.

    -

    Wrap-Up

    +

    Wrap-Up

    Events can come from the outside, but they can also be published externally—​our publish handler converts an event to a message on a Redis channel. We use events to talk to the outside world. This kind of temporal decoupling buys us a lot of flexibility in our application integrations, but -as always, it comes at a cost.

    +as always, it comes at a cost. +

    @@ -852,103 +1005,41 @@

    Wrap-Up

    -

    More generally, if you’re moving from a model of synchronous messaging to an +

    +More generally, if you’re moving from a model of synchronous messaging to an async one, you also open up a whole host of problems having to do with message -reliability and eventual consistency. Read on to [footguns].

    +reliability and eventual consistency. Read on to [footguns]. + + +

    +
    + -
    diff --git a/docs/book/chapter_12_cqrs.html b/book/chapter_12_cqrs.html similarity index 81% rename from docs/book/chapter_12_cqrs.html rename to book/chapter_12_cqrs.html index d34d2ce..329f3bb 100644 --- a/docs/book/chapter_12_cqrs.html +++ b/book/chapter_12_cqrs.html @@ -3,7 +3,7 @@ - + Command-Query Responsibility Segregation (CQRS) + @@ -81,7 +183,7 @@
    -

    Command-Query Responsibility Segregation (CQRS)

    +

    12: Command-Query Responsibility Segregation (CQRS)

    -

    In this chapter, we’re going to start with a fairly uncontroversial insight: +

    + + +In this chapter, we’re going to start with a fairly uncontroversial insight: reads (queries) and writes (commands) are different, so they should be treated differently (or have their responsibilities segregated, if you will). Then we’re going to push that insight as far as we can.

    @@ -156,9 +261,11 @@

    Command-Query Responsibility Segregation (CQRS)

    Figure 1. Separating reads from writes
    -

    Domain Models Are for Writing

    +

    Domain Models Are for Writing

    -

    We’ve spent a lot of time in this book talking about how to build software that +

    + +We’ve spent a lot of time in this book talking about how to build software that enforces the rules of our domain. These rules, or constraints, will be different for every application, and they make up the interesting core of our systems.

    @@ -177,7 +284,7 @@

    Domain Models Are for Writing

    def test_allocating_to_a_batch_reduces_the_available_quantity():
         batch = Batch("batch-001", "SMALL-TABLE", qty=20, eta=date.today())
    -    line = OrderLine('order-ref', "SMALL-TABLE", 2)
    +    line = OrderLine("order-ref", "SMALL-TABLE", 2)
     
         batch.allocate(line)
     
    @@ -187,7 +294,7 @@ 

    Domain Models Are for Writing

    def test_cannot_allocate_if_available_smaller_than_required(): small_batch, large_line = make_batch_and_line("ELEGANT-LAMP", 2, 20) - assert small_batch.can_allocate(large_line) is False
    + assert small_batch.can_allocate(large_line) is False
    @@ -211,9 +318,10 @@

    Domain Models Are for Writing

    -

    Most Users Aren’t Going to Buy Your Furniture

    +

    Most Users Aren’t Going to Buy Your Furniture

    -

    At MADE.com, we have a system very like the allocation service. In a busy day, we +

    +At MADE.com, we have a system very like the allocation service. In a busy day, we might process one hundred orders in an hour, and we have a big gnarly system for allocating stock to those orders.

    @@ -224,7 +332,9 @@

    Most Users Aren’t G us to deliver it.

    -

    The domain is the same—​we’re concerned with batches of stock, and their +

    + +The domain is the same—​we’re concerned with batches of stock, and their arrival date, and the amount that’s still available—​but the access pattern is very different. For example, our customers won’t notice if the query is a few seconds out of date, but if our allocate service is inconsistent, @@ -235,7 +345,9 @@

    Most Users Aren’t G
    Is Read Consistency Truly Attainable?
    -

    This idea of trading consistency against performance makes a lot of developers +

    + +This idea of trading consistency against performance makes a lot of developers nervous at first, so let’s talk quickly about that.

    @@ -286,7 +398,8 @@

    Most Users Aren’t G

    -

    We can think of these requirements as forming two halves of a system: +

    +We can think of these requirements as forming two halves of a system: the read side and the write side, shown in Read versus write.

    @@ -329,9 +442,13 @@

    Most Users Aren’t G

    -

    Post/Redirect/Get and CQS

    +

    Post/Redirect/Get and CQS

    -

    If you do web development, you’re probably familiar with the +

    + + + +If you do web development, you’re probably familiar with the Post/Redirect/Get pattern. In this technique, a web endpoint accepts an HTTP POST and responds with a redirect to see the result. For example, we might accept a POST to /batches to create a new batch and redirect the user to @@ -350,8 +467,8 @@

    Post/Redirect/Get and CQS

    write phases of our operation.

    -

    This technique is a simple example of command-query separation (CQS). In CQS we -follow one simple rule: functions should either modify state or answer +

    This technique is a simple example of command-query separation (CQS).[1] +We follow one simple rule: functions should either modify state or answer questions, but never both. This makes software easier to reason about: we should always be able to ask, "Are the lights on?" without flicking the light switch.

    @@ -372,28 +489,29 @@

    Post/Redirect/Get and CQS

    As you’ll see, we can use the CQS principle to make our systems faster and more -scalable, but first, let’s fix the CQS violation in our existing code. Ages ago, we introduced an allocate endpoint that takes an order and -calls our service layer to allocate some stock. At the end of the call, we -return a 200 OK and the batch ID. That’s led to some ugly design flaws so that -we can get the data we need. Let’s change it to return a simple OK message and -instead provide a new read-only endpoint to retrieve allocation state:

    +scalable, but first, let’s fix the CQS violation in our existing code. Ages +ago, we introduced an allocate endpoint that takes an order and calls our +service layer to allocate some stock. At the end of the call, we return a 200 +OK and the batch ID. That’s led to some ugly design flaws so that we can get +the data we need. Let’s change it to return a simple OK message and instead +provide a new read-only endpoint to retrieve allocation state:

    API test does a GET after the POST (tests/e2e/test_api.py)
    -
    @pytest.mark.usefixtures('postgres_db')
    -@pytest.mark.usefixtures('restart_api')
    +
    @pytest.mark.usefixtures("postgres_db")
    +@pytest.mark.usefixtures("restart_api")
     def test_happy_path_returns_202_and_batch_is_allocated():
         orderid = random_orderid()
    -    sku, othersku = random_sku(), random_sku('other')
    +    sku, othersku = random_sku(), random_sku("other")
         earlybatch = random_batchref(1)
         laterbatch = random_batchref(2)
         otherbatch = random_batchref(3)
    -    api_client.post_to_add_batch(laterbatch, sku, 100, '2011-01-02')
    -    api_client.post_to_add_batch(earlybatch, sku, 100, '2011-01-01')
    -    api_client.post_to_add_batch(otherbatch, othersku, 100, None)
    +    api_client.post_to_add_batch(laterbatch, sku, 100, "2011-01-02")
    +    api_client.post_to_add_batch(earlybatch, sku, 100, "2011-01-01")
    +    api_client.post_to_add_batch(otherbatch, othersku, 100, None)
     
         r = api_client.post_to_allocate(orderid, sku, qty=3)
         assert r.status_code == 202
    @@ -401,19 +519,19 @@ 

    Post/Redirect/Get and CQS

    r = api_client.get_allocation(orderid) assert r.ok assert r.json() == [ - {'sku': sku, 'batchref': earlybatch}, + {"sku": sku, "batchref": earlybatch}, ] -@pytest.mark.usefixtures('postgres_db') -@pytest.mark.usefixtures('restart_api') +@pytest.mark.usefixtures("postgres_db") +@pytest.mark.usefixtures("restart_api") def test_unhappy_path_returns_400_and_error_message(): unknown_sku, orderid = random_sku(), random_orderid() r = api_client.post_to_allocate( - orderid, unknown_sku, qty=20, expect_success=False, + orderid, unknown_sku, qty=20, expect_success=False ) assert r.status_code == 400 - assert r.json()['message'] == f'Invalid sku {unknown_sku}' + assert r.json()["message"] == f"Invalid sku {unknown_sku}" r = api_client.get_allocation(orderid) assert r.status_code == 404
    @@ -422,7 +540,9 @@

    Post/Redirect/Get and CQS

    -

    OK, what might the Flask app look like?

    +

    + +OK, what might the Flask app look like?

    Endpoint for viewing allocations (src/allocation/entrypoints/flask_app.py)
    @@ -432,12 +552,12 @@

    Post/Redirect/Get and CQS

    from allocation import views
     ...
     
    -@app.route("/allocations/<orderid>", methods=['GET'])
    +@app.route("/allocations/<orderid>", methods=["GET"])
     def allocations_view_endpoint(orderid):
         uow = unit_of_work.SqlAlchemyUnitOfWork()
         result = views.allocations(orderid, uow)  #(1)
         if not result:
    -        return 'not found', 404
    +        return "not found", 404
         return jsonify(result), 200
    @@ -454,9 +574,12 @@

    Post/Redirect/Get and CQS

    -

    Hold On to Your Lunch, Folks

    +

    Hold On to Your Lunch, Folks

    -

    Hmm, so we can probably just add a list method to our existing repository +

    + + +Hmm, so we can probably just add a list method to our existing repository object:

    @@ -466,17 +589,20 @@

    Hold On to Your Lunch, Folks

    from allocation.service_layer import unit_of_work
     
    +
     def allocations(orderid: str, uow: unit_of_work.SqlAlchemyUnitOfWork):
         with uow:
    -        results = list(uow.session.execute(
    -            'SELECT ol.sku, b.reference'
    -            ' FROM allocations AS a'
    -            ' JOIN batches AS b ON a.batch_id = b.id'
    -            ' JOIN order_lines AS ol ON a.orderline_id = ol.id'
    -            ' WHERE ol.orderid = :orderid',
    -            dict(orderid=orderid)
    -        ))
    -    return [{'sku': sku, 'batchref': batchref} for sku, batchref in results]
    + results = uow.session.execute( + """ + SELECT ol.sku, b.reference + FROM allocations AS a + JOIN batches AS b ON a.batch_id = b.id + JOIN order_lines AS ol ON a.orderline_id = ol.id + WHERE ol.orderid = :orderid + """, + dict(orderid=orderid), + ) + return [{"sku": sku, "batchref": batchref} for sku, batchref in results]
    @@ -518,9 +644,12 @@

    Hold On to Your Lunch, Folks

    -

    Testing CQRS Views

    +

    Testing CQRS Views

    -

    Before we get into exploring various options, let’s talk about testing. +

    + + +Before we get into exploring various options, let’s talk about testing. Whichever approaches you decide to go for, you’re probably going to need at least one integration test. Something like this:

    @@ -531,18 +660,18 @@

    Testing CQRS Views

    def test_allocations_view(sqlite_session_factory):
         uow = unit_of_work.SqlAlchemyUnitOfWork(sqlite_session_factory)
    -    messagebus.handle(commands.CreateBatch('sku1batch', 'sku1', 50, None), uow)  #(1)
    -    messagebus.handle(commands.CreateBatch('sku2batch', 'sku2', 50, today), uow)
    -    messagebus.handle(commands.Allocate('order1', 'sku1', 20), uow)
    -    messagebus.handle(commands.Allocate('order1', 'sku2', 20), uow)
    +    messagebus.handle(commands.CreateBatch("sku1batch", "sku1", 50, None), uow)  #(1)
    +    messagebus.handle(commands.CreateBatch("sku2batch", "sku2", 50, today), uow)
    +    messagebus.handle(commands.Allocate("order1", "sku1", 20), uow)
    +    messagebus.handle(commands.Allocate("order1", "sku2", 20), uow)
         # add a spurious batch and order to make sure we're getting the right ones
    -    messagebus.handle(commands.CreateBatch('sku1batch-later', 'sku1', 50, today), uow)
    -    messagebus.handle(commands.Allocate('otherorder', 'sku1', 30), uow)
    -    messagebus.handle(commands.Allocate('otherorder', 'sku2', 10), uow)
    +    messagebus.handle(commands.CreateBatch("sku1batch-later", "sku1", 50, today), uow)
    +    messagebus.handle(commands.Allocate("otherorder", "sku1", 30), uow)
    +    messagebus.handle(commands.Allocate("otherorder", "sku2", 10), uow)
     
    -    assert views.allocations('order1', uow) == [
    -        {'sku': 'sku1', 'batchref': 'sku1batch'},
    -        {'sku': 'sku2', 'batchref': 'sku2batch'},
    +    assert views.allocations("order1", uow) == [
    +        {"sku": "sku1", "batchref": "sku1batch"},
    +        {"sku": "sku2", "batchref": "sku2batch"},
         ]
    @@ -559,9 +688,12 @@

    Testing CQRS Views

    -

    "Obvious" Alternative 1: Using the Existing Repository

    +

    "Obvious" Alternative 1: Using the Existing Repository

    -

    How about adding a helper method to our products repository?

    +

    + + +How about adding a helper method to our products repository?

    A simple view that uses the repository (src/allocation/views.py)
    @@ -631,9 +763,11 @@

    "Obvious" Alternat

    -

    Your Domain Model Is Not Optimized for Read Operations

    +

    Your Domain Model Is Not Optimized for Read Operations

    -

    What we’re seeing here are the effects of having a domain model that +

    + +What we’re seeing here are the effects of having a domain model that is designed primarily for write operations, while our requirements for reads are often conceptually quite different.

    @@ -670,9 +804,12 @@

    Your Domain Mod

    -

    "Obvious" Alternative 2: Using the ORM

    +

    "Obvious" Alternative 2: Using the ORM

    -

    You may be thinking, OK, if our repository is clunky, and working with +

    + + +You may be thinking, OK, if our repository is clunky, and working with Products is clunky, then I can at least use my ORM and work with Batches. That’s what it’s for!

    @@ -691,7 +828,7 @@

    "Obvious" Alternative 2: Using the model.OrderLine.orderid == orderid ) return [ - {'sku': b.sku, 'batchref': b.batchref} + {"sku": b.sku, "batchref": b.batchref} for b in batches ]

    @@ -709,10 +846,12 @@

    "Obvious" Alternative 2: Using the
    -

    SELECT N+1 and Other Performance Considerations

    +

    SELECT N+1 and Other Performance Considerations

    -

    The so-called -SELECT N+1 +

    + + +The so-called SELECT N+1 problem is a common performance problem with ORMs: when retrieving a list of objects, your ORM will often perform an initial query to, say, get all the IDs of the objects it needs, and then issue individual queries for each object to @@ -727,9 +866,10 @@

    SELECT N+1 and Other Pe In all fairness, we should say that SQLAlchemy is quite good at avoiding the SELECT N+1 problem. It doesn’t display it in the preceding example, and - you can request - eager loading + you can request eager loading explicitly to avoid it when dealing with joined objects. + + @@ -744,9 +884,11 @@

    SELECT N+1 and Other Pe

    -

    Time to Completely Jump the Shark

    +

    Time to Completely Jump the Shark

    -

    On that note: have we convinced you that our raw SQL version isn’t so weird as +

    + +On that note: have we convinced you that our raw SQL version isn’t so weird as it first seemed? Perhaps we were exaggerating for effect? Just you wait.

    @@ -760,10 +902,12 @@

    Time to Completely Jump the Shark
    def allocations(orderid: str, uow: unit_of_work.SqlAlchemyUnitOfWork):
         with uow:
    -        results = list(uow.session.execute(
    -            'SELECT sku, batchref FROM allocations_view WHERE orderid = :orderid',
    -            dict(orderid=orderid)
    -        ))
    +        results = uow.session.execute(
    +            """
    +            SELECT sku, batchref FROM allocations_view WHERE orderid = :orderid
    +            """,
    +            dict(orderid=orderid),
    +        )
             ...

    @@ -778,10 +922,11 @@

    Time to Completely Jump the Shark
    allocations_view = Table(
    -    'allocations_view', metadata,
    -    Column('orderid', String(255)),
    -    Column('sku', String(255)),
    -    Column('batchref', String(255)),
    +    "allocations_view",
    +    metadata,
    +    Column("orderid", String(255)),
    +    Column("sku", String(255)),
    +    Column("batchref", String(255)),
     )
    @@ -797,7 +942,8 @@

    Time to Completely Jump the SharkSELECT * from mytable WHERE key = :value.

    -

    More than raw speed, though, this approach buys us scale. When we’re writing +

    +More than raw speed, though, this approach buys us scale. When we’re writing data to a relational database, we need to make sure that we get a lock over the rows we’re changing so we don’t run into consistency problems.

    @@ -822,13 +968,16 @@

    Time to Completely Jump the Shark
    -

    Keeping the read model up to date is the challenge! Database views +

    + + +Keeping the read model up to date is the challenge! Database views (materialized or otherwise) and triggers are a common solution, but that limits you to your database. We’d like to show you how to reuse our event-driven architecture instead.

    -

    Updating a Read Model Table Using an Event Handler

    +

    Updating a Read Model Table Using an Event Handler

    We add a second handler to the Allocated event:

    @@ -840,7 +989,7 @@

    Updating a Read Mod
    EVENT_HANDLERS = {
         events.Allocated: [
             handlers.publish_allocated_event,
    -        handlers.add_allocation_to_read_model
    +        handlers.add_allocation_to_read_model,
         ],

    @@ -856,13 +1005,16 @@

    Updating a Read Mod
    
     def add_allocation_to_read_model(
    -        event: events.Allocated, uow: unit_of_work.SqlAlchemyUnitOfWork,
    +    event: events.Allocated,
    +    uow: unit_of_work.SqlAlchemyUnitOfWork,
     ):
         with uow:
             uow.session.execute(
    -            'INSERT INTO allocations_view (orderid, sku, batchref)'
    -            ' VALUES (:orderid, :sku, :batchref)',
    -            dict(orderid=event.orderid, sku=event.sku, batchref=event.batchref)
    +            """
    +            INSERT INTO allocations_view (orderid, sku, batchref)
    +            VALUES (:orderid, :sku, :batchref)
    +            """,
    +            dict(orderid=event.orderid, sku=event.sku, batchref=event.batchref),
             )
             uow.commit()
    @@ -889,12 +1041,15 @@

    Updating a Read Mod ... def remove_allocation_from_read_model( - event: events.Deallocated, uow: unit_of_work.SqlAlchemyUnitOfWork, + event: events.Deallocated, + uow: unit_of_work.SqlAlchemyUnitOfWork, ): with uow: uow.session.execute( - 'DELETE FROM allocations_view ' - ' WHERE orderid = :orderid AND sku = :sku', + """ + DELETE FROM allocations_view + WHERE orderid = :orderid AND sku = :sku + ... @@ -956,7 +1111,9 @@

    Updating a Read Mod
    Rebuilding from Scratch
    -

    "What happens when it breaks?" should be the first question we ask as engineers.

    +

    + +"What happens when it breaks?" should be the first question we ask as engineers.

    How do we deal with a view model that hasn’t been updated because of a bug or @@ -979,7 +1136,7 @@

    Updating a Read Mod allocated

  • -

    Calls the add_allocate_to_read_model handler for each allocated item

    +

    Calls the add_allocation_to_read_model handler for each allocated item

  • @@ -992,9 +1149,11 @@

    Updating a Read Mod

    -

    Changing Our Read Model Implementation Is Easy

    +

    Changing Our Read Model Implementation Is Easy

    -

    Let’s see the flexibility that our event-driven model buys us in action, +

    + +Let’s see the flexibility that our event-driven model buys us in action, by seeing what happens if we ever decide we want to implement a read model by using a totally separate storage engine, Redis.

    @@ -1009,8 +1168,9 @@

    Changing Our Read Model
    def add_allocation_to_read_model(event: events.Allocated, _):
         redis_eventpublisher.update_readmodel(event.orderid, event.sku, event.batchref)
     
    +
     def remove_allocation_from_read_model(event: events.Deallocated, _):
    -    redis_eventpublisher.update_readmodel(event.orderid, event.sku, None)
    + redis_eventpublisher.update_readmodel(event.orderid, event.sku, None)

    @@ -1044,10 +1204,10 @@

    Changing Our Read Model
    -
    def allocations(orderid):
    +
    def allocations(orderid: str):
         batches = redis_eventpublisher.get_readmodel(orderid)
         return [
    -        {'batchref': b.decode(), 'sku': s.decode()}
    +        {"batchref": b.decode(), "sku": s.decode()}
             for s, b in batches.items()
         ]
    @@ -1070,6 +1230,7 @@

    Changing Our Read Model Event handlers are a great way to manage updates to a read model, if you decide you need one. They also make it easy to change the implementation of that read model at a later date. + @@ -1090,12 +1251,15 @@

    Changing Our Read Model

    -

    Wrap-Up

    +

    Wrap-Up

    -

    Trade-offs of various view model options proposes some pros and cons for each of our options.

    +

    + +Trade-offs of various view model options proposes some pros and cons for each of our options.

    -

    As it happens, the allocation service at MADE.com does use "full-blown" CQRS, +

    +As it happens, the allocation service at MADE.com does use "full-blown" CQRS, with a read model stored in Redis, and even a second layer of cache provided by Varnish. But its use cases are quite a bit different from what we’ve shown here. For the kind of allocation service we’re building, it seems @@ -1132,13 +1296,20 @@

    Wrap-Up

    Adds another query language with its own quirks and syntax.

    -

    Use hand-rolled SQL

    +

    Use hand-rolled SQL to query your normal model tables

    Offers fine control over performance with a standard query syntax.

    Changes to DB schema have to be made to your hand-rolled queries and your ORM definitions. Highly normalized schemas may still have performance limitations.

    +

    Add some extra (denormalized) tables to your DB as a read model

    +

    A denormalized table can be much faster to query. If we update the + normalized and denormalized ones in the same transaction, we will + still have good guarantees of data consistency

    +

    It will slow down writes slightly

    + +

    Create separate read stores with events

    Read-only copies are easy to scale out. Views can be constructed when data changes so that queries are as simple as possible.

    @@ -1161,101 +1332,42 @@

    Wrap-Up

    the chapter.

    -

    On that note, let’s sally forth into our final chapter.

    +

    On that note, let’s sally forth into our final chapter. +

    +
    + + +
    +
    +
    +1. We’re using the terms somewhat interchangeably, but CQS is normally something you apply to a single class or module: functions that read state should be separate from those that modify it. And CQRS is something you apply to your whole application: the classes, modules, code paths and even databases that read state can be separated from the ones that modify it.
    -
    diff --git a/docs/book/chapter_13_dependency_injection.html b/book/chapter_13_dependency_injection.html similarity index 81% rename from docs/book/chapter_13_dependency_injection.html rename to book/chapter_13_dependency_injection.html index dc4c1b4..17d2620 100644 --- a/docs/book/chapter_13_dependency_injection.html +++ b/book/chapter_13_dependency_injection.html @@ -3,7 +3,7 @@ - + Dependency Injection (and Bootstrapping) + @@ -81,7 +183,7 @@
    -

    Dependency Injection (and Bootstrapping)

    +

    13: Dependency Injection (and Bootstrapping)

    -

    Dependency injection (DI) is regarded with suspicion in the Python world. And +

    +Dependency injection (DI) is regarded with suspicion in the Python world. And we’ve managed just fine without it so far in the example code for this book!

    @@ -120,7 +223,9 @@

    Dependency Injection (and Bootstrapping for how to do it, leaving it to you to pick which you think is most Pythonic.

    -

    We’ll also add a new component to our architecture called bootstrap.py; +

    + +We’ll also add a new component to our architecture called bootstrap.py; it will be in charge of dependency injection, as well as some other initialization stuff that we often need. We’ll explain why this sort of thing is called a composition root in OO languages, and why bootstrap script is just fine @@ -188,9 +293,10 @@

    Dependency Injection (and Bootstrapping
    Figure 2. Bootstrap takes care of all that in one place

    -

    Implicit Versus Explicit Dependencies

    +

    Implicit Versus Explicit Dependencies

    -

    Depending on your particular brain type, you may have a slight +

    +Depending on your particular brain type, you may have a slight feeling of unease at the back of your mind at this point. Let’s bring it out into the open. We’ve shown you two ways of managing dependencies and testing them.

    @@ -206,7 +312,8 @@

    Implicit Versus Explicit Depende
    def allocate(
    -        cmd: commands.Allocate, uow: unit_of_work.AbstractUnitOfWork
    +    cmd: commands.Allocate,
    +    uow: unit_of_work.AbstractUnitOfWork,
     ):
    @@ -236,7 +343,6 @@

    Implicit Versus Explicit Depende
    class SqlAlchemyUnitOfWork(AbstractUnitOfWork):
    -
         def __init__(self, session_factory=DEFAULT_SESSION_FACTORY):
             self.session_factory = session_factory
             ...
    @@ -269,9 +375,11 @@

    Implicit Versus Explicit Depende

    -

    Aren’t Explicit Dependencies Totally Weird and Java-y?

    +

    Aren’t Explicit Dependencies Totally Weird and Java-y?

    -

    If you’re used to the way things normally happen in Python, you’ll be thinking +

    + +If you’re used to the way things normally happen in Python, you’ll be thinking all this is a bit weird. The standard way to do things is to declare our dependency implicitly by simply importing it, and then if we ever need to change it for tests, we can monkeypatch, as is Right and True in dynamic @@ -286,11 +394,12 @@

    Aren’t Expl ... def send_out_of_stock_notification( - event: events.OutOfStock, uow: unit_of_work.AbstractUnitOfWork, + event: events.OutOfStock, + uow: unit_of_work.AbstractUnitOfWork, ): email.send( #(2) - 'stock@made.com', - f'Out of stock for {event.sku}', + "stock@made.com", + f"Out of stock for {event.sku}", )

    @@ -307,7 +416,8 @@

    Aren’t Expl

    -

    Why pollute our application code with unnecessary arguments just for the +

    +Why pollute our application code with unnecessary arguments just for the sake of our tests? mock.patch makes monkeypatching nice and easy:

    @@ -342,7 +452,9 @@

    Aren’t Expl manage.

    -

    On top of that, declaring an explicit dependency is an example of the +

    + +On top of that, declaring an explicit dependency is an example of the dependency inversion principle—rather than having an (implicit) dependency on a specific detail, we have an (explicit) dependency on an abstraction:

    @@ -362,11 +474,12 @@

    Aren’t Expl
    def send_out_of_stock_notification(
    -        event: events.OutOfStock, send_mail: Callable,
    +    event: events.OutOfStock,
    +    send_mail: Callable,
     ):
         send_mail(
    -        'stock@made.com',
    -        f'Out of stock for {event.sku}',
    +        "stock@made.com",
    +        f"Out of stock for {event.sku}",
         )
    @@ -381,12 +494,16 @@

    Aren’t Expl pass them on?

    -

    That’s extra (duplicated) cruft for Flask, Redis, and our tests. Moreover, -putting all the responsibility for passing dependencies to the right handler -onto the message bus feels like a violation of the SRP.

    +

    It needs to happen as early as possible in the process lifecycle, so the most +obvious place is in our entrypoints. That would mean extra (duplicated) cruft +in Flask and Redis, and in our tests. And we’d also have to add the +responsibility for passing dependencies around to the message bus, which +already has a job to do; it feels like a violation of the SRP.

    -

    Instead, we’ll reach for a pattern called Composition Root (a bootstrap +

    + +Instead, we’ll reach for a pattern called Composition Root (a bootstrap script to you and me),[1] and we’ll do a bit of "manual DI" (dependency injection without a framework). See Bootstrapper between entrypoints and message bus.[2]

    @@ -427,9 +544,12 @@

    Aren’t Expl

    -

    Preparing Handlers: Manual DI with Closures and Partials

    +

    Preparing Handlers: Manual DI with Closures and Partials

    -

    One way to turn a function with dependencies into one that’s ready to be +

    + + +One way to turn a function with dependencies into one that’s ready to be called later with those dependencies already injected is to use closures or partial functions to compose the function with its dependencies:

    @@ -440,7 +560,8 @@

    Preparing Hand
    # existing allocate function, with abstract uow dependency
     def allocate(
    -        cmd: commands.Allocate, uow: unit_of_work.AbstractUnitOfWork
    +    cmd: commands.Allocate,
    +    uow: unit_of_work.AbstractUnitOfWork,
     ):
         line = OrderLine(cmd.orderid, cmd.sku, cmd.qty)
         with uow:
    @@ -474,9 +595,10 @@ 

    Preparing Hand
  • The difference between closures (lambdas or named functions) and functools.partial is that the former use -late -binding of variables, which can be a source of confusion if -any of the dependencies are mutable.

    +late binding of variables, +which can be a source of confusion if any of the dependencies are mutable. + +

  • @@ -490,10 +612,11 @@

    Preparing Hand
    -

    An Alternative Using Classes

    +

    An Alternative Using Classes

    -

    Closures and partial functions will feel familiar to people who’ve done a bit +

    + +Closures and partial functions will feel familiar to people who’ve done a bit of functional programming. Here’s an alternative using classes, which may appeal to others. It requires rewriting all our handler functions as classes, though:

    @@ -524,7 +649,6 @@

    An Alternative Using Classes

    # we replace the old `def allocate(cmd, uow)` with:
     
     class AllocateHandler:
    -
         def __init__(self, uow: unit_of_work.AbstractUnitOfWork):  #(2)
             self.uow = uow
     
    @@ -563,13 +687,15 @@ 

    An Alternative Using Classes

    -

    Use whichever you and your team feel more comfortable with.

    +

    +Use whichever you and your team feel more comfortable with.

    -

    A Bootstrap Script

    +

    A Bootstrap Script

    -

    We want our bootstrap script to do the following:

    +

    +We want our bootstrap script to do the following:

      @@ -596,7 +722,7 @@

      A Bootstrap Script

      def bootstrap(
      -    start_orm: bool = True,  #(1)
      +    start_orm: bool = True,  #(1)
           uow: unit_of_work.AbstractUnitOfWork = unit_of_work.SqlAlchemyUnitOfWork(),  #(2)
           send_mail: Callable = email.send,
           publish: Callable = redis_eventpublisher.publish,
      @@ -605,7 +731,7 @@ 

      A Bootstrap Script

      if start_orm: orm.start_mappers() #(1) - dependencies = {'uow': uow, 'send_mail': send_mail, 'publish': publish} + dependencies = {"uow": uow, "send_mail": send_mail, "publish": publish} injected_event_handlers = { #(3) event_type: [ inject_dependencies(handler, dependencies) @@ -631,8 +757,9 @@

      A Bootstrap Script

      1. orm.start_mappers() is our example of initialization work that needs -to be done once at the beginning of an app. We also see things like -setting up the logging module.

        +to be done once at the beginning of an app. Another common example is +setting up the logging module. +

      2. We can use the argument defaults to define what the normal/production @@ -650,7 +777,8 @@

        A Bootstrap Script

      -

      Here’s how we inject dependencies into a handler function by inspecting +

      +Here’s how we inject dependencies into a handler function by inspecting it:

      @@ -687,11 +815,13 @@

      A Bootstrap Script

      Even-More-Manual DI with Less Magic
      -

      If you’re finding the preceding inspect code a little harder to grok, this +

      +If you’re finding the preceding inspect code a little harder to grok, this even simpler version may appeal to you.

      -

      Harry wrote the code for inject_dependencies() as a first cut of how to do +

      +Harry wrote the code for inject_dependencies() as a first cut of how to do "manual" dependency injection, and when he saw it, Bob accused him of overengineering and writing his own DI framework.

      @@ -715,12 +845,11 @@

      A Bootstrap Script

      ], events.OutOfStock: [ lambda e: handlers.send_out_of_stock_notification(e, send_mail) - ] + ], } injected_command_handlers = { commands.Allocate: lambda c: handlers.allocate(c, uow), - commands.CreateBatch: \ - lambda c: handlers.add_batch(c, uow), + commands.CreateBatch: lambda c: handlers.add_batch(c, uow), commands.ChangeBatchQuantity: \ lambda c: handlers.change_batch_quantity(c, uow), }
      @@ -729,11 +858,11 @@

      A Bootstrap Script

      -

      Harry says he couldn’t even imagine writing out that many lines of code -and having to look up that many function arguments manually. -This is a perfectly viable solution, though, since it’s only one -line of code or so per handler you add, and thus not a massive maintenance burden -even if you have dozens of handlers.

      +

      Harry says he couldn’t even imagine writing out that many lines of code and +having to look up that many function arguments manually. It would be a +perfectly viable solution, though, since it’s only one line of code or so per +handler you add. Even if you have dozens of handlers, it wouldn’t be much of +maintenance burden.

      Our app is structured in such a way that we always want to do dependency @@ -741,24 +870,27 @@

      A Bootstrap Script

      and Harry’s inspect()-based one will both work fine.

      -

      If you find yourself wanting to do DI in more things and at different times, +

      + +If you find yourself wanting to do DI in more things and at different times, or if you ever get into dependency chains (in which your dependencies have their own dependencies, and so on), you may get some mileage out of a "real" DI framework.

      At MADE, we’ve used Inject in a few places, -and it’s fine, although it makes Pylint unhappy. You might also check out +and it’s fine (although it makes Pylint unhappy). You might also check out Punq, as written by Bob himself, or the -DRY-Python crew’s dependencies.

      +DRY-Python crew’s Dependencies.

    -

    Message Bus Is Given Handlers at Runtime

    +

    Message Bus Is Given Handlers at Runtime

    -

    Our message bus will no longer be static; it needs to have the already-injected +

    +Our message bus will no longer be static; it needs to have the already-injected handlers given to it. So we turn it from being a module into a configurable class:

    @@ -768,7 +900,6 @@

    Message Bus Is Given Handlers
    class MessageBus:  #(1)
    -
         def __init__(
             self,
             uow: unit_of_work.AbstractUnitOfWork,
    @@ -788,7 +919,7 @@ 

    Message Bus Is Given Handlers elif isinstance(message, commands.Command): self.handle_command(message) else: - raise Exception(f'{message} was not an Event or Command')

    + raise Exception(f"{message} was not an Event or Command")

    @@ -812,7 +943,11 @@

    Message Bus Is Given Handlers

    -

    What else changes in the bus?

    +

    + + + +What else changes in the bus?

    Event and command handler logic stays the same (src/allocation/service_layer/messagebus.py)
    @@ -822,22 +957,21 @@

    Message Bus Is Given Handlers
        def handle_event(self, event: events.Event):
             for handler in self.event_handlers[type(event)]:  #(1)
                 try:
    -                logger.debug('handling event %s with handler %s', event, handler)
    +                logger.debug("handling event %s with handler %s", event, handler)
                     handler(event)  #(2)
                     self.queue.extend(self.uow.collect_new_events())
                 except Exception:
    -                logger.exception('Exception handling event %s', event)
    +                logger.exception("Exception handling event %s", event)
                     continue
     
    -
         def handle_command(self, command: commands.Command):
    -        logger.debug('handling command %s', command)
    +        logger.debug("handling command %s", command)
             try:
                 handler = self.command_handlers[type(command)]  #(1)
                 handler(command)  #(2)
                 self.queue.extend(self.uow.collect_new_events())
             except Exception:
    -            logger.exception('Exception handling command %s', command)
    +            logger.exception("Exception handling command %s", command)
                 raise

    @@ -859,9 +993,11 @@

    Message Bus Is Given Handlers
    -

    Using Bootstrap in Our Entrypoints

    +

    Using Bootstrap in Our Entrypoints

    -

    In our application’s entrypoints, we now just call bootstrap.bootstrap() +

    + +In our application’s entrypoints, we now just call bootstrap.bootstrap() and get a message bus that’s ready to go, rather than configuring a UoW and the rest of it:

    @@ -873,20 +1009,20 @@

    Using Bootstrap in Our Entrypoints<
    -from allocation import views
     +from allocation import bootstrap, views
     
    - app = Flask(__name__)
    + app = Flask(__name__)
     -orm.start_mappers()  #(1)
     +bus = bootstrap.bootstrap()
     
     
    - @app.route("/add_batch", methods=['POST'])
    + @app.route("/add_batch", methods=["POST"])
     @@ -19,8 +16,7 @@ def add_batch():
    -     cmd = commands.CreateBatch(
    -         request.json['ref'], request.json['sku'], request.json['qty'], eta,
    -     )
    +     cmd = commands.CreateBatch(
    +         request.json["ref"], request.json["sku"], request.json["qty"], eta
    +     )
     -    uow = unit_of_work.SqlAlchemyUnitOfWork()  #(2)
     -    messagebus.handle(cmd, uow)
     +    bus.handle(cmd)  #(3)
    -     return 'OK', 201
    + return "OK", 201

    @@ -908,9 +1044,12 @@

    Using Bootstrap in Our Entrypoints<
    -

    Initializing DI in Our Tests

    +

    Initializing DI in Our Tests

    -

    In tests, we can use bootstrap.bootstrap() with overridden defaults to get a +

    + + +In tests, we can use bootstrap.bootstrap() with overridden defaults to get a custom message bus. Here’s an example in an integration test:

    @@ -918,24 +1057,25 @@

    Initializing DI in Our Tests

    -
    @pytest.fixture
    +
    @pytest.fixture
     def sqlite_bus(sqlite_session_factory):
         bus = bootstrap.bootstrap(
    -        start_orm=True,  #(1)
    +        start_orm=True,  #(1)
             uow=unit_of_work.SqlAlchemyUnitOfWork(sqlite_session_factory),  #(2)
    -        send_mail=lambda *args: None,  #(3)
    -        publish=lambda *args: None,  #(3)
    +        send_mail=lambda *args: None,  #(3)
    +        publish=lambda *args: None,  #(3)
         )
         yield bus
         clear_mappers()
     
    +
     def test_allocations_view(sqlite_bus):
    -    sqlite_bus.handle(commands.CreateBatch('sku1batch', 'sku1', 50, None))
    -    sqlite_bus.handle(commands.CreateBatch('sku2batch', 'sku2', 50, today))
    +    sqlite_bus.handle(commands.CreateBatch("sku1batch", "sku1", 50, None))
    +    sqlite_bus.handle(commands.CreateBatch("sku2batch", "sku2", 50, today))
         ...
    -    assert views.allocations('order1', sqlite_bus.uow) == [
    -        {'sku': 'sku1', 'batchref': 'sku1batch'},
    -        {'sku': 'sku2', 'batchref': 'sku2batch'},
    +    assert views.allocations("order1", sqlite_bus.uow) == [
    +        {"sku": "sku1", "batchref": "sku1batch"},
    +        {"sku": "sku2", "batchref": "sku2batch"},
         ]
    @@ -955,7 +1095,8 @@

    Initializing DI in Our Tests

    -

    In our unit tests, in contrast, we can reuse our FakeUnitOfWork:

    +

    +In our unit tests, in contrast, we can reuse our FakeUnitOfWork:

    Bootstrap in unit test (tests/unit/test_handlers.py)
    @@ -964,10 +1105,10 @@

    Initializing DI in Our Tests

    def bootstrap_test_app():
         return bootstrap.bootstrap(
    -        start_orm=False,  #(1)
    +        start_orm=False,  #(1)
             uow=FakeUnitOfWork(),  #(2)
    -        send_mail=lambda *args: None,  #(3)
    -        publish=lambda *args: None,  #(3)
    +        send_mail=lambda *args: None,  #(3)
    +        publish=lambda *args: None,  #(3)
         )
    @@ -1003,9 +1144,10 @@

    Initializing DI in Our Tests

    -

    Building an Adapter "Properly": A Worked Example

    +

    Building an Adapter "Properly": A Worked Example

    -

    To really get a feel for how it all works, let’s work through an example of how +

    +To really get a feel for how it all works, let’s work through an example of how you might "properly" build an adapter and do dependency injection for it.

    @@ -1061,9 +1203,11 @@

    Building an Adapter "Pro through how you might define a more complex dependency.

    -

    Define the Abstract and Concrete Implementations

    +

    Define the Abstract and Concrete Implementations

    -

    We’ll imagine a more generic notifications API. Could be +

    + +We’ll imagine a more generic notifications API. Could be email, could be SMS, could be Slack posts one day.

    @@ -1072,54 +1216,54 @@

    Define the Abstract a
    class AbstractNotifications(abc.ABC):
    -
    -    @abc.abstractmethod
    +    @abc.abstractmethod
         def send(self, destination, message):
             raise NotImplementedError
     
     ...
     
     class EmailNotifications(AbstractNotifications):
    -
         def __init__(self, smtp_host=DEFAULT_HOST, port=DEFAULT_PORT):
             self.server = smtplib.SMTP(smtp_host, port=port)
             self.server.noop()
     
         def send(self, destination, message):
    -        msg = f'Subject: allocation service notification\n{message}'
    +        msg = f"Subject: allocation service notification\n{message}"
             self.server.sendmail(
    -            from_addr='allocations@example.com',
    +            from_addr="allocations@example.com",
                 to_addrs=[destination],
    -            msg=msg
    +            msg=msg,
             )

    -

    We change the dependency in the bootstrap script:

    +

    +We change the dependency in the bootstrap script:

    Notifications in message bus (src/allocation/bootstrap.py)
    -

    Make a Fake Version for Your Tests

    +

    Make a Fake Version for Your Tests

    -

    We work through and define a fake version for unit testing:

    +

    +We work through and define a fake version for unit testing:

    Fake notifications (tests/unit/test_handlers.py)
    @@ -1127,7 +1271,6 @@

    Make a Fake Version for Your Tests<
    class FakeNotifications(notifications.AbstractNotifications):
    -
         def __init__(self):
             self.sent = defaultdict(list)  # type: Dict[str, List[str]]
     
    @@ -1149,15 +1292,15 @@ 

    Make a Fake Version for Your Tests<
        def test_sends_email_on_out_of_stock_error(self):
             fake_notifs = FakeNotifications()
             bus = bootstrap.bootstrap(
    -            start_orm=False,
    +            start_orm=False,
                 uow=FakeUnitOfWork(),
                 notifications=fake_notifs,
    -            publish=lambda *args: None,
    +            publish=lambda *args: None,
             )
    -        bus.handle(commands.CreateBatch("b1", "POPULAR-CURTAINS", 9, None))
    +        bus.handle(commands.CreateBatch("b1", "POPULAR-CURTAINS", 9, None))
             bus.handle(commands.Allocate("o1", "POPULAR-CURTAINS", 10))
    -        assert fake_notifs.sent['stock@made.com'] == [
    -            f"Out of stock for POPULAR-CURTAINS",
    +        assert fake_notifs.sent["stock@made.com"] == [
    +            f"Out of stock for POPULAR-CURTAINS",
             ]

    @@ -1165,9 +1308,10 @@

    Make a Fake Version for Your Tests<

    -

    Figure Out How to Integration Test the Real Thing

    +

    Figure Out How to Integration Test the Real Thing

    -

    Now we test the real thing, usually with an end-to-end or integration +

    +Now we test the real thing, usually with an end-to-end or integration test. We’ve used MailHog as a real-ish email server for our Docker dev environment:

    @@ -1176,40 +1320,41 @@

    Figure Out How to In
    -
    version: "3"
    +
    version: "3"
     
    -services:
    +services:
     
    -  redis_pubsub:
    -    build:
    -      context: .
    -      dockerfile: Dockerfile
    -    image: allocation-image
    -    ...
    +  redis_pubsub:
    +    build:
    +      context: .
    +      dockerfile: Dockerfile
    +    image: allocation-image
    +    ...
     
    -  api:
    -    image: allocation-image
    -    ...
    +  api:
    +    image: allocation-image
    +    ...
     
    -  postgres:
    -    image: postgres:9.6
    -    ...
    +  postgres:
    +    image: postgres:9.6
    +    ...
     
    -  redis:
    -    image: redis:alpine
    -    ...
    +  redis:
    +    image: redis:alpine
    +    ...
     
    -  mailhog:
    -    image: mailhog/mailhog
    -    ports:
    -      - "11025:1025"
    -      - "18025:8025"
    + mailhog: + image: mailhog/mailhog + ports: + - "11025:1025" + - "18025:8025"

    -

    In our integration tests, we use the real EmailNotifications class, +

    +In our integration tests, we use the real EmailNotifications class, talking to the MailHog server in the Docker cluster:

    @@ -1217,32 +1362,32 @@

    Figure Out How to In
    -
    @pytest.fixture
    +
    @pytest.fixture
     def bus(sqlite_session_factory):
         bus = bootstrap.bootstrap(
    -        start_orm=True,
    +        start_orm=True,
             uow=unit_of_work.SqlAlchemyUnitOfWork(sqlite_session_factory),
             notifications=notifications.EmailNotifications(),  #(1)
    -        publish=lambda *args: None,
    +        publish=lambda *args: None,
         )
         yield bus
         clear_mappers()
     
     
     def get_email_from_mailhog(sku):  #(2)
    -    host, port = map(config.get_email_host_and_port().get, ['host', 'http_port'])
    -    all_emails = requests.get(f'http://{host}:{port}/api/v2/messages').json()
    -    return next(m for m in all_emails['items'] if sku in str(m))
    +    host, port = map(config.get_email_host_and_port().get, ["host", "http_port"])
    +    all_emails = requests.get(f"http://{host}:{port}/api/v2/messages").json()
    +    return next(m for m in all_emails["items"] if sku in str(m))
     
     
     def test_out_of_stock_email(bus):
         sku = random_sku()
    -    bus.handle(commands.CreateBatch('batch1', sku, 9, None))  #(3)
    -    bus.handle(commands.Allocate('order1', sku, 10))
    +    bus.handle(commands.CreateBatch("batch1", sku, 9, None))  #(3)
    +    bus.handle(commands.Allocate("order1", sku, 10))
         email = get_email_from_mailhog(sku)
    -    assert email['Raw']['From'] == 'allocations@example.com'  #(4)
    -    assert email['Raw']['To'] == ['stock@made.com']
    -    assert f'Out of stock for {sku}' in email['Raw']['Data']
    + assert email["Raw"]["From"] == "allocations@example.com" #(4) + assert email["Raw"]["To"] == ["stock@made.com"] + assert f"Out of stock for {sku}" in email["Raw"]["Data"]
    @@ -1271,7 +1416,8 @@

    Figure Out How to In
    Exercise for the Reader 2
    -

    You could do two things for practice regarding adapters:

    +

    +You could do two things for practice regarding adapters:

      @@ -1292,13 +1438,15 @@

      Figure Out How to In

    -

    Wrap-Up

    +

    Wrap-Up

    • Once you have more than one adapter, you’ll start to feel a lot of pain from passing dependencies around manually, unless you do some kind of -dependency injection.

      +dependency injection. + +

    • Setting up dependency injection is just one of many typical @@ -1347,18 +1495,27 @@

      Wrap-Up

      Test the less fake "real" thing.

    • -

      Profit!

      +

      Profit! +

    -

    These were the last patterns we wanted to cover, which brings us to the end of [part2]. In the epilogue, we’ll try to give you some pointers for applying these techniques in the Real WorldTM.

    +

    These were the last patterns we wanted to cover, which brings us to the end of +[part2]. In the epilogue, we’ll +try to give you some pointers for applying these techniques in the Real +WorldTM.

    +

    @@ -1374,93 +1531,22 @@

    Wrap-Up

    -
    diff --git a/docs/book/colo.html b/book/colo.html similarity index 100% rename from docs/book/colo.html rename to book/colo.html diff --git a/docs/book/copyright.html b/book/copyright.html similarity index 100% rename from docs/book/copyright.html rename to book/copyright.html diff --git a/docs/book/cover.html b/book/cover.html similarity index 100% rename from docs/book/cover.html rename to book/cover.html diff --git a/docs/book/epilogue_1_how_to_get_there_from_here.html b/book/epilogue_1_how_to_get_there_from_here.html similarity index 93% rename from docs/book/epilogue_1_how_to_get_there_from_here.html rename to book/epilogue_1_how_to_get_there_from_here.html index 1c4355c..e41dec8 100644 --- a/docs/book/epilogue_1_how_to_get_there_from_here.html +++ b/book/epilogue_1_how_to_get_there_from_here.html @@ -3,7 +3,7 @@ - + Epilogue + @@ -81,7 +183,7 @@
    -

    Epilogue: Epilogue

    +

    Epilogue: Epilogue

    -

    What Now?

    +

    What Now?

    Phew! We’ve covered a lot of ground in this book, and for most of our audience all of these ideas are new. With that in mind, we can’t hope to make you experts @@ -131,7 +233,7 @@

    What Now?

    -

    How Do I Get There from Here?

    +

    How Do I Get There from Here?

    Chances are that a lot of you are thinking something like this:

    @@ -176,7 +278,7 @@

    How Do I Get There from Here?

    -

    Separating Entangled Responsibilities

    +

    Separating Entangled Responsibilities

    At the beginning of the book, we said that the main characteristic of a big ball of mud is homogeneity: every part of the system looks the same, because we @@ -367,7 +469,7 @@

    Separating Entangled Responsibil

    -

    Identifying Aggregates and Bounded Contexts

    +

    Identifying Aggregates and Bounded Contexts

    Part of the problem with the codebase in our case study was that the object graph was highly connected. Each account had many workspaces, and each workspace had @@ -667,7 +769,7 @@

    Identifying Aggregates and
    Tip
    -We use this same technique in [chapter_11_external_events], where we replace a +We use this same technique in [chapter_12_cqrs], where we replace a nested loop over ORM objects with a simple SQL query. It’s the first step in a CQRS approach. @@ -696,7 +798,7 @@

    Identifying Aggregates and

    -

    An Event-Driven Approach to Go to Microservices via Strangler Pattern

    +

    An Event-Driven Approach to Go to Microservices via Strangler Pattern

    The Strangler Fig pattern involves creating a new system around the edges of an old system, while keeping it running. Bits of old functionality @@ -868,7 +970,7 @@

    -

    Convincing Your Stakeholders to Try Something New

    +

    Convincing Your Stakeholders to Try Something New

    If you’re thinking about carving a new system out of a big ball of mud, you’re probably suffering problems with reliability, performance, maintainability, or @@ -933,7 +1035,7 @@

    Convincing Your Stak
    Tip
    -Check out www.eventmodeling.org and www.eventstorming.org for some great +Check out www.eventmodeling.org and www.eventstorming.com for some great guides to visual modeling of systems with events. @@ -1015,7 +1117,7 @@

    Convincing Your Stak

    -

    Questions Our Tech Reviewers Asked That We Couldn’t Work into Prose

    +

    Questions Our Tech Reviewers Asked That We Couldn’t Work into Prose

    Here are some questions we heard during drafting that we couldn’t find a good place to address elsewhere in the book:

    @@ -1039,12 +1141,12 @@

    Que

    Of course you can! The techniques we’re presenting in this book are intended to make your life easier. They’re not some kind of ascetic discipline with which to punish yourself.

    -

    In our first case-study system, we had a lot of View Builder objects that used repositories to fetch data and then performed some transformations to return dumb read models. The advantage is that when you hit a performance problem, it’s easy to rewrite a view builder to use custom queries or raw SQL.

    +

    In the workspace/documents case-study system, we had a lot of View Builder objects that used repositories to fetch data and then performed some transformations to return dumb read models. The advantage is that when you hit a performance problem, it’s easy to rewrite a view builder to use custom queries or raw SQL.

    How should use cases interact across a larger system? Is it a problem for one to call another?
    -

    This might be an interim step. Again, in the first case study, we had handlers that would need to invoke other handlers. This gets really messy, though, and it’s much better to move to using a message bus to separate these concerns.

    +

    This might be an interim step. Again, in the documents case study, we had handlers that would need to invoke other handlers. This gets really messy, though, and it’s much better to move to using a message bus to separate these concerns.

    Generally, your system will have a single message bus implementation and a bunch of subdomains that center on a particular aggregate or set of aggregates. When your use case has finished, it can raise an event, and a handler elsewhere can run.

    @@ -1114,7 +1216,7 @@

    Que

    -

    Footguns

    +

    Footguns

    OK, so we’ve given you a whole bunch of new toys to play with. Here’s the fine print. Harry and Bob do not recommend that you copy and paste our code into @@ -1172,7 +1274,7 @@

    Footguns

    -

    More Required Reading

    +

    More Required Reading

    A few more books we’d like to recommend to help you on your way:

    @@ -1195,7 +1297,7 @@

    More Required Reading

    -

    Wrap-Up

    +

    Wrap-Up

    Phew! That’s a lot of warnings and reading suggestions; we hope we haven’t scared you off completely. Our goal with this book is to give you @@ -1207,96 +1309,30 @@

    Wrap-Up

    + -
    diff --git a/docs/book/images/C4.puml b/book/images/C4.puml similarity index 100% rename from docs/book/images/C4.puml rename to book/images/C4.puml diff --git a/docs/book/images/C4_Component.puml b/book/images/C4_Component.puml similarity index 100% rename from docs/book/images/C4_Component.puml rename to book/images/C4_Component.puml diff --git a/docs/book/images/C4_Container.puml b/book/images/C4_Container.puml similarity index 100% rename from docs/book/images/C4_Container.puml rename to book/images/C4_Container.puml diff --git a/docs/book/images/C4_Context.puml b/book/images/C4_Context.puml similarity index 100% rename from docs/book/images/C4_Context.puml rename to book/images/C4_Context.puml diff --git a/docs/book/images/apwp_0001.png b/book/images/apwp_0001.png similarity index 100% rename from docs/book/images/apwp_0001.png rename to book/images/apwp_0001.png diff --git a/docs/book/images/apwp_0002.png b/book/images/apwp_0002.png similarity index 100% rename from docs/book/images/apwp_0002.png rename to book/images/apwp_0002.png diff --git a/docs/book/images/apwp_0101.png b/book/images/apwp_0101.png similarity index 100% rename from docs/book/images/apwp_0101.png rename to book/images/apwp_0101.png diff --git a/docs/book/images/apwp_0102.png b/book/images/apwp_0102.png similarity index 100% rename from docs/book/images/apwp_0102.png rename to book/images/apwp_0102.png diff --git a/docs/book/images/apwp_0103.png b/book/images/apwp_0103.png similarity index 100% rename from docs/book/images/apwp_0103.png rename to book/images/apwp_0103.png diff --git a/docs/book/images/apwp_0104.png b/book/images/apwp_0104.png similarity index 100% rename from docs/book/images/apwp_0104.png rename to book/images/apwp_0104.png diff --git a/docs/book/images/apwp_0201.png b/book/images/apwp_0201.png similarity index 100% rename from docs/book/images/apwp_0201.png rename to book/images/apwp_0201.png diff --git a/docs/book/images/apwp_0202.png b/book/images/apwp_0202.png similarity index 100% rename from docs/book/images/apwp_0202.png rename to book/images/apwp_0202.png diff --git a/docs/book/images/apwp_0203.png b/book/images/apwp_0203.png similarity index 100% rename from docs/book/images/apwp_0203.png rename to book/images/apwp_0203.png diff --git a/docs/book/images/apwp_0204.png b/book/images/apwp_0204.png similarity index 100% rename from docs/book/images/apwp_0204.png rename to book/images/apwp_0204.png diff --git a/docs/book/images/apwp_0205.png b/book/images/apwp_0205.png similarity index 100% rename from docs/book/images/apwp_0205.png rename to book/images/apwp_0205.png diff --git a/docs/book/images/apwp_0206.png b/book/images/apwp_0206.png similarity index 100% rename from docs/book/images/apwp_0206.png rename to book/images/apwp_0206.png diff --git a/docs/book/images/apwp_0301.png b/book/images/apwp_0301.png similarity index 100% rename from docs/book/images/apwp_0301.png rename to book/images/apwp_0301.png diff --git a/docs/book/images/apwp_0302.png b/book/images/apwp_0302.png similarity index 100% rename from docs/book/images/apwp_0302.png rename to book/images/apwp_0302.png diff --git a/docs/book/images/apwp_0401.png b/book/images/apwp_0401.png similarity index 100% rename from docs/book/images/apwp_0401.png rename to book/images/apwp_0401.png diff --git a/docs/book/images/apwp_0402.png b/book/images/apwp_0402.png similarity index 100% rename from docs/book/images/apwp_0402.png rename to book/images/apwp_0402.png diff --git a/docs/book/images/apwp_0403.png b/book/images/apwp_0403.png similarity index 100% rename from docs/book/images/apwp_0403.png rename to book/images/apwp_0403.png diff --git a/docs/book/images/apwp_0404.png b/book/images/apwp_0404.png similarity index 100% rename from docs/book/images/apwp_0404.png rename to book/images/apwp_0404.png diff --git a/docs/book/images/apwp_0405.png b/book/images/apwp_0405.png similarity index 100% rename from docs/book/images/apwp_0405.png rename to book/images/apwp_0405.png diff --git a/docs/book/images/apwp_0501.png b/book/images/apwp_0501.png similarity index 100% rename from docs/book/images/apwp_0501.png rename to book/images/apwp_0501.png diff --git a/docs/book/images/apwp_0601.png b/book/images/apwp_0601.png similarity index 100% rename from docs/book/images/apwp_0601.png rename to book/images/apwp_0601.png diff --git a/docs/book/images/apwp_0602.png b/book/images/apwp_0602.png similarity index 100% rename from docs/book/images/apwp_0602.png rename to book/images/apwp_0602.png diff --git a/docs/book/images/apwp_0701.png b/book/images/apwp_0701.png similarity index 100% rename from docs/book/images/apwp_0701.png rename to book/images/apwp_0701.png diff --git a/docs/book/images/apwp_0702.png b/book/images/apwp_0702.png similarity index 100% rename from docs/book/images/apwp_0702.png rename to book/images/apwp_0702.png diff --git a/docs/book/images/apwp_0703.png b/book/images/apwp_0703.png similarity index 100% rename from docs/book/images/apwp_0703.png rename to book/images/apwp_0703.png diff --git a/docs/book/images/apwp_0704.png b/book/images/apwp_0704.png similarity index 100% rename from docs/book/images/apwp_0704.png rename to book/images/apwp_0704.png diff --git a/docs/book/images/apwp_0705.png b/book/images/apwp_0705.png similarity index 100% rename from docs/book/images/apwp_0705.png rename to book/images/apwp_0705.png diff --git a/docs/book/images/apwp_0801.png b/book/images/apwp_0801.png similarity index 100% rename from docs/book/images/apwp_0801.png rename to book/images/apwp_0801.png diff --git a/docs/book/images/apwp_0901.png b/book/images/apwp_0901.png similarity index 100% rename from docs/book/images/apwp_0901.png rename to book/images/apwp_0901.png diff --git a/docs/book/images/apwp_0902.png b/book/images/apwp_0902.png similarity index 100% rename from docs/book/images/apwp_0902.png rename to book/images/apwp_0902.png diff --git a/docs/book/images/apwp_0903.png b/book/images/apwp_0903.png similarity index 100% rename from docs/book/images/apwp_0903.png rename to book/images/apwp_0903.png diff --git a/docs/book/images/apwp_0904.png b/book/images/apwp_0904.png similarity index 100% rename from docs/book/images/apwp_0904.png rename to book/images/apwp_0904.png diff --git a/docs/book/images/apwp_1101.png b/book/images/apwp_1101.png similarity index 100% rename from docs/book/images/apwp_1101.png rename to book/images/apwp_1101.png diff --git a/docs/book/images/apwp_1102.png b/book/images/apwp_1102.png similarity index 100% rename from docs/book/images/apwp_1102.png rename to book/images/apwp_1102.png diff --git a/docs/book/images/apwp_1103.png b/book/images/apwp_1103.png similarity index 100% rename from docs/book/images/apwp_1103.png rename to book/images/apwp_1103.png diff --git a/docs/book/images/apwp_1104.png b/book/images/apwp_1104.png similarity index 100% rename from docs/book/images/apwp_1104.png rename to book/images/apwp_1104.png diff --git a/docs/book/images/apwp_1105.png b/book/images/apwp_1105.png similarity index 100% rename from docs/book/images/apwp_1105.png rename to book/images/apwp_1105.png diff --git a/docs/book/images/apwp_1106.png b/book/images/apwp_1106.png similarity index 100% rename from docs/book/images/apwp_1106.png rename to book/images/apwp_1106.png diff --git a/docs/book/images/apwp_1201.png b/book/images/apwp_1201.png similarity index 100% rename from docs/book/images/apwp_1201.png rename to book/images/apwp_1201.png diff --git a/docs/book/images/apwp_1202.png b/book/images/apwp_1202.png similarity index 100% rename from docs/book/images/apwp_1202.png rename to book/images/apwp_1202.png diff --git a/docs/book/images/apwp_1301.png b/book/images/apwp_1301.png similarity index 100% rename from docs/book/images/apwp_1301.png rename to book/images/apwp_1301.png diff --git a/docs/book/images/apwp_1302.png b/book/images/apwp_1302.png similarity index 100% rename from docs/book/images/apwp_1302.png rename to book/images/apwp_1302.png diff --git a/docs/book/images/apwp_1303.png b/book/images/apwp_1303.png similarity index 100% rename from docs/book/images/apwp_1303.png rename to book/images/apwp_1303.png diff --git a/docs/book/images/apwp_aa01.png b/book/images/apwp_aa01.png similarity index 100% rename from docs/book/images/apwp_aa01.png rename to book/images/apwp_aa01.png diff --git a/docs/book/images/apwp_ep01.png b/book/images/apwp_ep01.png similarity index 100% rename from docs/book/images/apwp_ep01.png rename to book/images/apwp_ep01.png diff --git a/docs/book/images/apwp_ep02.png b/book/images/apwp_ep02.png similarity index 100% rename from docs/book/images/apwp_ep02.png rename to book/images/apwp_ep02.png diff --git a/docs/book/images/apwp_ep03.png b/book/images/apwp_ep03.png similarity index 100% rename from docs/book/images/apwp_ep03.png rename to book/images/apwp_ep03.png diff --git a/docs/book/images/apwp_ep04.png b/book/images/apwp_ep04.png similarity index 100% rename from docs/book/images/apwp_ep04.png rename to book/images/apwp_ep04.png diff --git a/docs/book/images/apwp_ep05.png b/book/images/apwp_ep05.png similarity index 100% rename from docs/book/images/apwp_ep05.png rename to book/images/apwp_ep05.png diff --git a/docs/book/images/apwp_ep06.png b/book/images/apwp_ep06.png similarity index 100% rename from docs/book/images/apwp_ep06.png rename to book/images/apwp_ep06.png diff --git a/docs/book/images/apwp_p101.png b/book/images/apwp_p101.png similarity index 100% rename from docs/book/images/apwp_p101.png rename to book/images/apwp_p101.png diff --git a/docs/book/images/apwp_p201.png b/book/images/apwp_p201.png similarity index 100% rename from docs/book/images/apwp_p201.png rename to book/images/apwp_p201.png diff --git a/docs/book/images/cover.png b/book/images/cover.png similarity index 100% rename from docs/book/images/cover.png rename to book/images/cover.png diff --git a/docs/book/introduction.html b/book/introduction.html similarity index 85% rename from docs/book/introduction.html rename to book/introduction.html index 313f5bd..00e80d0 100644 --- a/docs/book/introduction.html +++ b/book/introduction.html @@ -3,7 +3,7 @@ - + Introduction + @@ -81,7 +183,7 @@
    -

    Introduction

    +

    Introduction

    -

    Why Do Our Designs Go Wrong?

    +

    Why Do Our Designs Go Wrong?

    What comes to mind when you hear the word chaos? Perhaps you think of a noisy stock exchange, or your kitchen in the morning—​everything confused and @@ -137,7 +239,7 @@

    Why Do Our Designs Go Wrong?

    classes that perform no calculations but do perform I/O; and everything coupled to everything else so that changing any part of the system becomes fraught with danger. This is so common that software engineers have their own term for -chaos: the Big Ball of Mud anti-pattern (A real-life dependency diagram (source: "Enterprise Dependency: Big Ball of Yarn" by Alex Papadimoulis)).

    +chaos: the Big Ball of Mud antipattern (A real-life dependency diagram (source: "Enterprise Dependency: Big Ball of Yarn" by Alex Papadimoulis)).

    @@ -164,7 +266,7 @@

    Why Do Our Designs Go Wrong?

    -

    Encapsulation and Abstractions

    +

    Encapsulation and Abstractions

    Encapsulation and abstraction are tools that we all instinctively reach for as programmers, even if we don’t all use these exact words. Allow us to dwell @@ -189,7 +291,7 @@

    Encapsulation and Abstractions

    from urllib.request import urlopen from urllib.parse import urlencode -params = dict(q='Sausages', format='json') +params = dict(q='Sausages', format='json') handle = urlopen('http://api.duckduckgo.com' + '?' + urlencode(params)) raw_text = handle.read().decode('utf8') parsed = json.loads(raw_text) @@ -197,7 +299,7 @@

    Encapsulation and Abstractions

    results = parsed['RelatedTopics'] for r in results: if 'Text' in r: - print(r['FirstURL'] + ' - ' + r['Text'])
    + print(r['FirstURL'] + ' - ' + r['Text'])
    @@ -209,13 +311,13 @@

    Encapsulation and Abstractions

    import requests
     
    -params = dict(q='Sausages', format='json')
    +params = dict(q='Sausages', format='json')
     parsed = requests.get('http://api.duckduckgo.com/', params=params).json()
     
     results = parsed['RelatedTopics']
     for r in results:
         if 'Text' in r:
    -        print(r['FirstURL'] + ' - ' + r['Text'])
    + print(r['FirstURL'] + ' - ' + r['Text'])
    @@ -231,13 +333,13 @@

    Encapsulation and Abstractions

    it explicit:

    -
    Do a search with the duckduckgo module
    +
    Do a search with the duckduckgo client library
    -
    import duckduckgo
    -for r in duckduckgo.query('Sausages').results:
    -    print(r.url + ' - ' + r.text)
    +
    import duckduckpy
    +for r in duckduckpy.query('Sausages').related_topics:
    +    print(r.first_url, ' - ', r.text)
    @@ -285,7 +387,7 @@

    Encapsulation and Abstractions

    -

    Layering

    +

    Layering

    Encapsulation and abstraction help us by hiding details and protecting the consistency of our data, but we also need to pay attention to the interactions @@ -295,7 +397,7 @@

    Layering

    In a big ball of mud, the dependencies are out of control (as you saw in -A real-life dependency diagram (source: "Enterprise Dependency: Big Ball of Yarn" by Alex Papadimoulis)). Changing one node of the graph becomes difficult because it +A real-life dependency diagram (source: "Enterprise Dependency: Big Ball of Yarn" by Alex Papadimoulis)). Changing one node of the graph becomes difficult because it has the potential to affect many other parts of the system. Layered architectures are one way of tackling this problem. In a layered architecture, we divide our code into discrete categories or roles, and we introduce rules @@ -343,7 +445,7 @@

    Layering

    -

    The Dependency Inversion Principle

    +

    The Dependency Inversion Principle

    You might be familiar with the dependency inversion principle (DIP) already, because it’s the D in SOLID.[2]

    @@ -436,7 +538,7 @@

    The Dependency Inversion Principle

    -

    A Place for All Our Business Logic: The Domain Model

    +

    A Place for All Our Business Logic: The Domain Model

    But before we can turn our three-layered architecture inside out, we need to talk more about that middle layer: the high-level modules or business @@ -453,6 +555,11 @@

    A Place for All Ou

    +

    @@ -465,93 +572,22 @@

    A Place for All Ou

    -
    diff --git a/docs/book/ix.html b/book/ix.html similarity index 100% rename from docs/book/ix.html rename to book/ix.html diff --git a/docs/book/part1.html b/book/part1.html similarity index 80% rename from docs/book/part1.html rename to book/part1.html index b80987e..bae1485 100644 --- a/docs/book/part1.html +++ b/book/part1.html @@ -3,7 +3,7 @@ - + Building an Architecture to Support Domain Modeling