BROUGHT TO YOU IN PARTNERSHIP WITH

API Integration
CONTENTS

∙ Introduction
∙ API Integration Practices
∙ Integration Practice Matrix
∙ API Integration Patterns

Practices and Patterns

‑ Error Codes
‑ Authentication
‑ Eventing
‑ Querying
‑ Pagination
‑ Bulk
∙ Additional Resources

UPDATED BY BRIAN BUSCH

PRODUCT MARKETING & ALLIANCES, CLOUD ELEMENTS

We all hear it so often that we almost stop hearing it: “Integration instance of systems or applications you control — vs. digital
is critical to meeting users’ needs.” products, which offer integration as a feature or service in support
of products’ value propositions.
Integration work consumes 50%-80% of the time and budget of
digital transformation projects, or building a digital platform, while Beyond that, the explosion of applications used today means
innovation gets only the leftovers, according to SAP and Salesforce. there’s a ton of integrations to build, so think about the integration
And as everyone from legacy enterprises to SaaS startups launches experience you need to offer or who will do the building. Will it be
new digital products, they all hit a point at which the product centralized (in IT) and/or productized (for self-serve use), or will it
cannot unlock more value for users or continue to grow without be user-driven (usually for ad hoc integration needs). With these you
making integration a feature. can create a simple two-by-two matrix (see Figure 1 on page three).

If I were to sum up the one question behind all of the other questions As they say, there’s a right tool for every job — in this case, an
that I hear from customers, enterprises, partners, and developers, integration platform or iPaaS. Let’s consider some common
it would be something like: “Is integration a differentiator that we integration use cases, and then look at the pros and cons of these
should own? Or an undifferentiated but necessary feature that options.
supports what we’re trying to accomplish?”

This Refcard won’t try to answer that question for you. Rather, no
matter what type of development work you do, API integration is a
fact of life today, like gravity. Why? Today, experience is paramount.
The average enterprise uses more than 1,500 cloud applications
(with the number growing by 20% each year). Every app needs to
integrate with other systems in a fluid and ever-changing application
ecosystem. So instead, I’ll share some of the common practices
you’re likely to contend with as well as some patterns to consider.

API INTEGRATION PRACTICES

Gartner will tell you there are a dozen or more integration
scenarios or needs, but when you squint at all of them, a bigger
distinction stands out: internal integration — connecting your

1
 REFCARD | API INTEGRATION PRACTICES AND PATTERNS

Figure 1: Integration practice matrix

HYPERSCALER DEPLOYMENT AND/OR CLOUD-TO-CLOUD INTEGRATION FOR

CLOUD DATA LAKE DIGITAL PRODUCTS
Moving applications and data from on-prem architectures to cloud Enterprises and startups alike generally build customer-facing
services, whether to save costs or better analyze that data, typically applications in the cloud with modern, RESTful APIs. However, every
requires re-architecting the stack itself. So it’s likely that IT or the API implements standards like OAuth differently, and there’s always
CIO’s office will also need to centralize the integration work to handle data mapping and transformation work to be done for point-to-point
the complexity. But if you had an integration platform with truly integrations. Not to mention customers rarely have the time and
robust API connectors (e.g., scheduled bulk operations, webhooks, skills to do the integration work themselves.
polling, etc.), could IT build reusable components that ad hoc
A bolt-on iPaaS can be a cheap, quick way to ease the burden
integrators could use themselves?
for users, but then you’re limited by the iPaaS UX and their API
connectors. Consider options that allow you to embed one-to-many
GROUND-TO-CLOUD INTEGRATION FOR
PROCESS IMPROVEMENT integrations within your product UI/UX via API call — you don’t have
Perhaps your company’s enterprise resource planning (ERP) or to build or maintain the underlying code, and you get to own the
finance and accounting (F&A) system runs on-prem still, but the user experience.
CFO’s office wants to take advantage of a cloud-based accounts
payable (AP) automation platform like Tipalti, Tungsten Networks, or CUSTOM APPLICATION-TO-CLOUD INTEGRATION

Coupa? You likely have a legacy enterprise service bus (ESB) or other Let’s say that your company has, or is building, custom applications

middleware running on-prem and integrated with the ERP, but it will and needs to integrate them with other cloud (or on-prem)

take a lot of custom code to work with Tipalti’s API. applications and data. No iPaaS will offer an API connector to your
custom app out of the box, which means you’ll have to write and
If Tipalti offers the embedded integration for your ERP, problem maintain custom code, perhaps within a legacy iPaaS.
solved. Otherwise, look for an API-accessible, pre-built connector like
those from SAP Open Connectors or Axway’s Integration Builder.

3 BROUGHT TO YOU IN PARTNERSHIP WITH

 REFCARD | API INTEGRATION PRACTICES AND PATTERNS

Instead, look for an extensible integration platform that allows you Table 1: Error code descriptions
to create reusable, first-class API connectors that others (developers,
CODE ERROR AND DESCRIPTION
ad hoc integrators, etc.) across the company could take advantage
of to deliver their own integrations and take pressure off the 1XX Informational 302 Temp. Found 410 Gone

centralized integration team.

2XX Success 303 See Other 411 Length Req.

API INTEGRATION PATTERNS 3XX Redirection 304 Not Modified 412 Precondition
Integration is more complex than it looks. It’s often said that
connecting to an API is easy, but secure integrations that deliver 4XX Client Error 307 Temp. Redirect 413 Entity Too Big

seamless, effortless, and highly performant user experiences are

5XX Server Error 400 Bad Request 414 URI Too Long
hard. Why?

• Every API is unique – Like snowflakes, 200 Executed 401 Unauthorized 415 No Media Type

researching and building integrations means

201 Created 403 Forbidden 417 Failed
peeling back layers of nuance, including SOAP vs.
REST, XML vs. JSON, different auth mechanisms, 202 Accepted 404 Not Found 500 Internal Error
workarounds for migrations when <5% of APIs
offer bulk data operations, webhooks vs. polling 204 No Content 405 Not a Method 501 Not Implemented

for eventing, unique error codes, limited search

301 Permanent Move 406 Not Accepted 503 Unavailable
and discovery mechanisms, etc.

• Every data model is unique – This requires

AUTHENTICATION
developers to solve complex data mapping and
Getting the right access to the right data underpins any integration
transformation problems for every integration.
project, yet authentication can be one of the hardest parts.
• Every workflow is different – From operations Authentication at its core is the ability to prove your application’s
on the data itself to lookups and contingent identity. There are several different ways applications can grant
logic, developers need the right tools to not only access to developers to create integrations.
connect systems, but to also improve — and even
automate — otherwise manual process steps. BASIC CREDENTIALS
This approach is listed for educational purposes but is, fortunately,
With that in mind, below are some of the nuanced issues that becoming less common due to its security vulnerability of man-in-
developers encounter daily as they build and maintain integrations. the-middle attacks. This method uses the username and password in
the HTTP header when you are making calls, so the security becomes
ERROR CODES
dependent on the availability of SSL. This method lends itself to
When creating integrations, it’s important to understand what error
internal APIs for resources that aren’t generally exposed elsewhere or
codes you might be getting from the application provider. As you
for IoT projects.
start sending requests, it’s helpful to understand when things work,
but it becomes equally important to understand why they don’t. API KEYS
The use of API keys is still pervasive and very common for API access.
With integrations, error codes resulting from a lack of access to size
It is a fast, easy way to authenticate accounts and is relatively low
limitations help guide your application’s business logic as well. While
overhead for the provider, as the provider can easily create and
many error codes are not normalized, Table 1 is a general reference
purge API keys. API keys are a uniquely generated set of characters,
guide you can use when working with REST APIs (see next column).
sometimes random, and are often sent as a pair, a user, and a secret.

When receiving API keys, it’s best to copy and store them in a
password manager and treat them like any other password.

4 BROUGHT TO YOU IN PARTNERSHIP WITH

 REFCARD | API INTEGRATION PRACTICES AND PATTERNS

OAUTH 2.0 WEBHOOKS

OAuth is a bit different in that it is token-based. There are three Webhooks allow you to get updates as they occur in real time, as they
major components in OAuth: the user, the consumer or integrating are pull-based instead of push-based. In this method, when updates
application, and the service provider. In this flow, the user grants the occur from the application, they are sent to a URL that you’ve
consumer (i.e., the application you want to be integrated) access to specified for your application to use. This push-based refreshing of
the service provider by an exchange of tokens through the OAuth information is what gives applications the ability to update in real
endpoint before accessing data from the API. time and create dynamic user experiences.

Figure 2: OAuth workflow Additionally, it is the preferred pattern since the implementation is
just a URL, as opposed to polling where you need to create a polling
framework that manages the frequency and scope of the information
you wish to receive. For example, Marketo’s polling framework is 239
lines of code compared to just five lines for a similar webhook.

Figure 3: Polling vs. webhooks

OAuth is the preferred and best-practice approach to authentication

because it allows users to decide what level of access the integrating
application can have while also being able to set time-based limits.
Some APIs have shorter time limits for which the use of refresh
tokens is needed.
QUERYING

EVENTING We’ve automated the movement of data so that our integration is live

We’ve navigated documentation, authenticated access, and can /GET and fluid, but now it’s flowing in like a river when all we really need

a “Hello World” response — now, let’s bring the data to life, or at least is a small piece of the data that is useful to what our application is

automate it. In some cases, eventing isn’t needed and you can skip doing. We need to set query parameters to filter out all of the data

ahead. However, the true power of integration is the ability to put that we don’t need. Querying is the last part of the API endpoint, or

data in motion without the bottleneck of human clicks. There are two path, that directs the call to pull only the data you want:

primary ways to automate or “event” your API calls.

GET /widgets?query_parameter_here

POLLING
The ability to search against databases as well as the target
Polling is very much like the kid sitting in the backseat on a road trip
applications is a crucial step. This allows you to test and confirm
constantly asking, “Are we there yet?” But in our case, computers
responses and see any difference in the data structure between what
don’t get annoyed, so the conversation goes like this:
is returned and what is in the documentation. Querying allows you to
“Any changes in data yet?” modify the key-value pairs of the request. At the end of the URL path,
“No, no, no, no, yes, no, no, yes, and so on.”
the query starts with a question mark (?) and is separated with an
To detect if there have been any changes in the data — create, ampersand (&). For example:
retrieve, and delete events, specifically — polling is sending requests
GET /widgets?type=best&date=yesterday
at a predetermined frequency and waiting for the endpoint to
respond. If there is no response body, then there have been no
To test an endpoint, there are multiple options depending on which
changes. Because of this continual request for information, polling
language you prefer, but the majority of API documentation will
is very inefficient and can be expensive when paying for computing
reference commands in curl. “cURL” stands for Client URL and
time, as the vast majority of requests go unanswered.
is a command line tool that becomes particularly valuable when
Finally, because of the intervals in timing, polling is immediately retrieving data from other applications. To test an API, open up your
out of date once the API call returns, batching any new updates that Terminal and copy/paste the curl command to see what response
would have happened since the last call. you get back. Simple enough, right?

5 BROUGHT TO YOU IN PARTNERSHIP WITH

 REFCARD | API INTEGRATION PRACTICES AND PATTERNS

Now try adjusting the key-value pairs and fine tune to just the FIXED DATA PAGES
information your application needs from the target endpoint. When Fairly straightforward, when adding into the query, you select
adding parameters to a curl command directly, be sure to add a which page of data you would like returned. For example, to
backslash (/) before the question mark of the query as ? and & return the fourth page of results, you would use this query: GET /
are special characters. Tweaking these pairs will decrease the size widgets?page=4. This method is preferred if the application you
and overhead your application might expect. Simplicity is the act are integrating with has known pages and you would like the data
of demystifying something without losing the magic, and the same returned sequentially. However, it becomes harder if you aren't sure
could be said for APIs. You can always increase the amount of data — what exactly is on that fourth page.
especially in testing — to see where you might run into errors.
FLEXIBLE DATA PAGES
SORTING AND ORDERING Similar to fixed data pages, you could still call the fourth page
An important piece to getting the information you need is how the of widgets, but now you can specify the size of the page. Calling
data will be presented at first glance, which becomes especially GET /widgets?page=4&page_size=25 allows you to further
helpful in testing and presenting data through a UI. Many APIs will dictate what you will get back in terms of page size. This can be very
support sort, sort_by, or order parameters in the URL to change helpful if you're building a UI and need the results to be a certain size.
the ascending or descending nature of the data you’re working with.
For example, a GET /widgets?sort_by=desc(created) can give BULK
us the freshest widgets in inventory. Up to this point, API integration has focused on specific sets of
data, but what happens when you need to move a large amount of
PAGINATION data from one system to another? Some applications expose this
There might be way too much data, or data that just keeps going ability with Bulk APIs. Bulk APIs allow you to update, insert, and
and scrolling. This is where pagination comes in. Pagination is the delete a large number of records all at once. This can be particularly
ability to put that giant pool of responses from every customer useful when transferring large systems of record (e.g., marketing
you’ve had since 1994 into human-readable pages without seizing automation, CRM, or ERP) from one provider to another.
up your computer. Pagination requires some implied order by a
Bulk actually operates in several batches of data sets when a request
field, or fields, like unique id, created date, modified date, and so on.
for a bulk job is sent to the application provider. The application
There are a few different types of pagination you might encounter.
provider will send batches over asynchronously to complete the
OFFSET job. Depending on the size of the data set, the job will also send you
This is easiest to implement and is therefore very common. a unique identifier to check the job status and close the job once
The LIMIT is the number of rows that will be returned, and complete. Be sure to double-check the file type that a bulk API will
OFFSET is the starting point of results. For example, a path of provide — either CSV, JSON, or XML.
/widgets?limit=10 would return 10 rows for the first page. Then
Additionally, if you’re not getting the full data set back, be sure to
the second page could be /widgets?limit=10&offset=10.
check any API rate limits that apply, as you may exceed them with
This would return 10 rows, starting with the 10 row, and so on. The
th large data transfers.
downside of using offset pagination is it becomes less performant
Figure 4: Bulk data transfer
with large data sets. For example, if the offset is 1M rows, it will still
have to run through 1M rows before returning the limit requested.

KEYSET
A keyset is helpful to get around large data sets and uses the filter
value of the last page of results as the starting point for the next page
of results using an additional limit URL parameter. For example, the
first call could be GET /widgets?limit=10 with either a unique
identifier like date created or modified. The second call would be
GET /widgets?limit=10&created:lte:2019-09, the next would
be GET /widgets?limit=10&created:lte:2019-10, and so on.

6 BROUGHT TO YOU IN PARTNERSHIP WITH

 REFCARD | API INTEGRATION PRACTICES AND PATTERNS

ADDITIONAL RESOURCES
*Books from the “Foundations of RESTful Architecture” Refcard:
• Richardson, L., & Amundsen, M. (2015). RESTful web APIs.
O’Reilly. WRITTEN BY BRIAN BUSCH,
• Allamaraju, S. (2010). RESTful web services cookbook. PRODUCT MARKETING & ALLIANCES,
CLOUD ELEMENTS
O’Reilly.
• Masse, M. (2011). REST API design rulebook: Designing Brian Busch is an API enthusiast, leads
consistent RESTful web service interfaces. O’Reilly. Product and Alliances Marketing for Cloud
Elements, and has been involved in launching
• Biehl, M. (2016). RESTful API design. CreateSpace. and scaling new products and services throughout his
• Webber, J., Parastatidis, S., & Robinson, I. (2010). REST in career, most recently with Kapost (now Upland Kapost) and
practice. O’Reilly. before that, Captricity (now Vidado.ai).

• Louvel, J. (2013). Restlet in action: Developing RESTful web He holds degrees from Boston College and an MBA from UC
APIs in Java. Manning. Berkeley-Haas. @brbusch
• Kalali, M., & Mehta, B. (2013). Developing RESTful services
with JAX-RS 2.0, WebSockets, and JSON. Packt Publishing.

DZone, a Devada Media Property, is the resource software Devada, Inc.

developers, engineers, and architects turn to time and 600 Park Offices Drive
again to learn new skills, solve software development Suite 150
problems, and share their expertise. Every day, hundreds of Research Triangle Park, NC 27709
thousands of developers come to DZone to read about the
888.678.0399 919.678.0300
latest technologies, methodologies, and best practices. That
makes DZone the ideal place for developer marketers to Copyright © 2020 Devada, Inc. All rights reserved. No part
build product and brand awareness and drive sales. DZone of this publication may be reproduced, stored in a retrieval
clients include some of the most innovative technology and system, or transmitted, in any form or by means of electronic,
tech-enabled companies in the world including Red Hat, mechanical, photocopying, or otherwise, without prior
Cloud Elements, Sensu, and Sauce Labs. written permission of the publisher.

7 BROUGHT TO YOU IN PARTNERSHIP WITH