I've worked on two different products that provided API design guide tooling. The idea for both was the same—provide a tool that helps companies design consistent APIs and helps communicate good design patterns to all of their developers.

It's common for a company to take inventory of their APIs and see lots of inconsistency. One API uses camel case while another uses snake case. One puts the version in the URL, another doesn't. These companies try to address this by putting together their own set of API guidelines and adopting tools that help them apply these guidelines to their APIs during the development process. This is where our design guide tooling came in.

Design guide tooling—which goes by names like style guide, standardization, governance, and linting—allows people to express guidelines in code and apply those guidelines to an OpenAPI document. Imagine a rule that makes sure every API endpoint requires authentication and returns a 403 when not provided. When a rule like this fails, the tools can prevent the teams from merging the code until they resolve the issue. It’s a big win for scaling an API strategy across many teams.

But there's always been a nagging feeling something is missing.

Seeing through the lens of Optic#

For one, over time you start to see that a diff of an OpenAPI file is not the same as an API changelog. This isn’t apparent when first designing an API, but it starts to show as you evolve an API over time.

For example, when someone opens a PR with a change to an OpenAPI file, the Git diff shows which lines in the file were affected. It’s up to the reviewer to decipher what it meant when the author removed line 20. If it meant a field was removed, what sections of the API are affected? Is it a breaking change? Does it only affect a single response or does it affect several API endpoints? Traditional design guide tooling can’t give this insight and understanding because it lacks the history of how the API has changed. An OpenAPI diff is not a changelog.

Second, over time you start to wish you could write guidelines around how APIs change, not just around the correctness of an OpenAPI document. It only takes one time pushing a change to production with unintended consequences to feel this.

For instance, if someone adds a required field to an existing API endpoint, it would be helpful to get immediate and automatic feedback that the change might break API clients. But traditional design guide tooling isn’t tracking how an API is changing, so people can’t write guidelines that give insights and feedback about API changes. The tooling can only look at a snapshot in time.

I’ve wanted something in the past to address these nagging feelings but was limited by the traditional tooling and approaches. Since viewing API development through the lens of Optic, though, I’ve been able to see how it can address these issues. Optic records the entire history of an API as people evolve it over time, similar to how Git records all of the changes to your code. And with this detailed history, Optic can provide insights into API changes and give people the ability to write guidelines around changes that go beyond OpenAPI diffs.

Designing how APIs change#

When we design an API, one thing we’re doing is designing how the API will change over time. The way we design change influences the way we design other aspects of the API, such as the behavior or the technical details.

For instance, when we talk about versioning an API, we're talking about how we intend to treat breaking changes. When we decide to always use a JSON object with our API responses, we're making room for that response to grow over time. When we say we won't add a required field to an existing endpoint, we're making a promise to people using the API that we won't introduce certain kinds of change.

There are best practices for dealing with change and evolving APIs, but they require a human to know them and spot any issues. They have to decide on their own if a change will cause something to break. This takes a lot of time—for many it's a full time job.

With some of the new tools at Optic, we want to empower developers to make good decisions that have a positive impact on their API consumers. We want to give them insights into how their API is changing so it’s easier to do the right thing than it is to do the wrong one.

Dealing with the noise#

Most of the existing design guide tools are too noisy to be useful.

Linting tools for instance run sets of rules against an entire OpenAPI document and give feedback when something doesn't pass a rule. This is helpful for new API designs. It gives the team a list of issues it can correct before releasing the API to the public.

But for existing APIs, this feedback can be overwhelming. Running a linter on an API that has never had linting could produce dozens of errors and warnings that the team may be unable to fix. Over time, this turns into noise that the team ignores—a list of 100 linting errors is just as helpful as zero. Would someone notice if it found error 101? Would the team be able to fix all 101 errors without breaking your consumers?

Focusing the feedback on the latest API changes helps limit the feedback to what’s important now, not feedback on the entire API definition. For instance, a tool like Optic can provide feedback in a Pull Request (PR) based on what's changed along with any recommendations for improvement. This makes the recommendations more actionable in the moment.

Starting discussions#

Giving recommendations is one thing. Getting teams to talk about those recommendations is another. This is important to help teams build consistent APIs.

Many companies have shifted away from strict API governance to focusing on creating a culture around API excellence. Adding tools is easy, but changing culture and garnering adoption requires time, effort, and a commitment to a long-term process.

The goal is to focus on the culture around collaboration, knowledge sharing, training, and discussions. It's about creating a space where people can have autonomy while getting expert guidance on their work.

We're trying to line up with this approach. We want to help start conversations. We want guidelines that are less about pointing out what's wrong with an OpenAPI document and more about getting people talking about the way an API is evolving. We know from experience that this approach works.

Making it a reality#

We're working on the design guide tools and services right now that give people visibility over how things change. We want to share with you our focus and what we're thinking.

• Dealing with the noise. We want to give insights to what is changing in a PR. An API may have 20 endpoints, but we'll only focus on what’s changing in the PR. This allows the design guides work on existing APIs.
• Starting conversations. We want to loop people into the conversation based on the change. If someone wants to join discussions when people add new endpoints, Optic can make sure they get notified.
• Writing guidelines around change. We want to enable people to write guidelines focused on change. Whether it's looking at a new status code, an updated field in a response body, or a whole new endpoint, we want to enable people to write rules that have not been possible before.

We believe our approach lends itself well to these workflows. Optic helps teams keep their API specification up-to-date by observing API traffic and making it easy to update the API specification when they change the API. Our GitBot adds an API Changelog to every PR that updates the API, creating the context for these kinds of conversations, and for automated, actionable feedback from tooling.

Sharing your stories#

We want to hear about your experiences changing APIs. What were the struggles? Fill out the survey below and we’ll help you write some of the first Optic powered design guides over the next several weeks.

Once our APIs are deployed in a real environment like staging or production, traffic races in from all directions, interacting with the API in the ways we expect, and in ways we could never anticipate. These 'surprises' can teach us things and help us build better designed and more robust APIs.

Physicists smash particles into one another in giant particle accelerators looking for surprising patterns, and for new information that questions their assumptions about reality.

When we put our APIs in the path of real traffic, reality smashes into our API specifications, our well-landscaped-happy-paths, and our assumptions. We learn, revise, and improve our work -- hopefully.

There is so much to be learned from looking at our API's actual behavior and usage in the real-world. But are we really looking? Can we tell if our APIs are working as expected from the logs we collect today?

Users have long asked for options to run Optic in real environments to get more observability into their API's real behavior. We have even observed teams using the local-cli in their real environments without official tooling or support from the main project. Now you can join the beta and start monitoring your deployed APIs with Optic. You can also check out the docs to learn more about how the integrations work.

API Observability#

Most API logs contain only the URL, method, status code, timing information, and some open-tracing telemetry. This log level is like a magnifying glass, but to understand our API's real-world behavior we need a microscope. With deeper observability, you begin to tackle a new set of questions and use insights from real environments to make better decisions throughout the API lifecycle.

• Is the API working as designed? Is it meeting its contract?
• Are there additional fields (maybe even some sensitive ones) being returned?
• How are users interacting with our API? What patterns, flows and use cases are most important?
• Are there endpoints we did not document or test that users are working with?
• Who are our consumers? What subset of our API does each of them use?

Challenges meet Open Source Solutions#

Today's API logs are limited for good reasons: logging API bodies in a privacy/security conscious way is a hard problem. How do you log enough data to detect errant API behavior or security issues without also logging PII or sensitive information? tl;dr Sanitizing is hard.

We need a new standard for API logs, something privacy/security friendly by default, that can tell us much more about our API's real-world behavior than our current logging approaches.

Optic built and open sourced the shape-hash which logs only the schema/shape of your API bodies, but not the actual user data. If you have sent the Optic contributors a Debug Capture you have already used shape-hash.

Now we're doubling down and collaborating with a team of Cloud Provider/APM/Logging veterans to develop the next version of our API logging format. It's a much more powerful and secure open logging format for APIs that we hope the industry can adopt as a standard. Stay tuned for more information

Get Documentation and Observability for Every API#

Have an existing API that needs to be documented? Or 10? Or 100?

With a few hours of monitoring real API behavior, Optic can help you rapidly document any API you point it at. Deploy Optic monitoring, go to sleep, come back a day later, and your API will be accurately documented.

Once the API is documented, Optic will keep monitoring the API behavior to ensure every API change gets reviewed and approved before getting deployed.

Use Real Traffic for Contract Testing#

When Optic runs in Staging or QA environments, it reports on the API behavior of every deployment. All the traffic from these environments is monitored and helps test your API contract. It's like running the api status command, but for all the traffic captured in the real environment.

Everything your team already does in these environments now becomes part of how your API is tested: manual tests, automated tests, integration tests, and even the traffic from developers using the service while programming.

By taking all this test traffic into account, API test coverage increases substantially, without requiring your team to write or maintain tests by hand.

Map Dependencies and Manage Changes#

By watching how all your services interact, Optic can map the dependencies between them. This is invaluable when trying to understand the impact of any API change or planning sunsets / deprecations.

Imagine if this information was available to you when you were designing APIs or talking about changes to existing services?

• Will anyone be affected by changing an endpoint? Who?
• Which APIs are no longer used? Are they safe to delete?
• What request parameters do users send? What request parameters are no longer used?

Future versions of Optic's GitBot will make these insights available at design time and when talking about API changes in code review: i.e. "This will be a breaking change for teams x,y,z's usage pattern!".

[Insert your Magical Use Case]#

What questions do you have about how your APIs are consumed? What tools, alerts or reports would you build if your logs had better API observability?

We want to hear from you. Feel free to start a discussion on Optic's GitHub

Use Optic to Monitor APIs in Real Environments#

Interested in running Optic in your real environment? Want to be part of designing how we build API observability into the Optic project? Join the beta ⤵️

Sign up for the beta#

Optic ensures every change to your APIs gets documented, reviewed and approved before it is merged. By observing traffic that is sent to your API, Optic informs you of any changes it sees in behavior, allowing you to keep the specification up to date. If you run IntelliJ IDEA, you can integrate Optic into your workflow and always catch the latest changes as you work. It runs alongside your existing project configuration, in any run mode including debugging.

To everyone who has tried Optic, contributed code, told us your stories, written about the project, and helped us chart our course — we (the maintainers) want to say thank you! We know it’s been a crazy year and API tools are not the most important happening in the world right now. Still you’ve made the time for our open source project, and in doing so, helped keep us all employed. We can’t thank you enough for giving our team the opportunity to work on these problems. Thank you, sincerely.

Today I’m really excited to show off a new ‘primitive’ for API tooling that Optic has been working on. Everyone, meet spec coverage.

API spec coverage measures how much of your API spec was covered by traffic, and like everything else we do, it’s open source!

The Optic CLI now produces a coverage report for any of your test scripts, Postman collections, or even a bunch of manual CURLs to a server.

Optic helps teams write and maintain their first OpenAPI specifications. You don't need to get your team on-board to learn OpenAPI or worry about maintaining 10k line YAML files -- Optic takes care of all of that. Optic manages this process whether your development environment is running locally on your machine out in the Cloud, or somewhere in between. No matter where you develop and test, you have confidence that every change to your API is reviewed, approved, and documented before it's released. Optic can work at any interface in your project, even with microservices.

Optic helps teams write and maintain their first OpenAPI specifications. You don't need to get your team on-board to learn OpenAPI or worry about maintaining 10k line YAML files -- Optic takes care of all of that. Optic manages this process whether your development environment is running locally on your machine or out in the Cloud. No matter where you develop and test, you have confidence that every change to your API is reviewed, approved, and documented before it's released. Your documentation can provide details for your procedures, and assist in your governance goals.

Part 1 of our Changelog Specification Series#

We've had several other API tools engage us about using the Optic Changelog Spec in their products so we're going back to the basics with a repost from our July 8th, 2019 Problems with API Specifications. This is where Dev and I first shared Changelogs specs publicly :) Enjoy!

It's been over a year since Dev first came up with the idea to apply CQRS / Event Sourcing to modeling API behavior, and the idea has proven to be very robust. Since publishing the original form of this article, Optic has pivoted from a GUI for API contracts, to a code generator, then to an API spec generator, and finally to the "Git for APIs" tool that it is today. Don't worry -- you've shown us it's working, and we're sticking with it. No big pivots ahead.

This past summer, I worked with the Optic team as a software engineering intern. While it's the first time I've worked in an all-remote environment, I had an amazing experience and learned a lot from everyone here.

I've gotten to poke around the entire codebase, talk to all the amazing team members here, and work on everything from marketing to devops to even working on new products.

Optic is a Git-like version control system for API contracts. With Optic, maintaining an accurate API contract is as easy as reviewing diffs and approving suggested changes. This proves itself to be an incredibly valuable tool, and I spent a lot of this summer using Optic hands on.

Optic helps teams write and maintain their first OpenAPI specifications. You don't need to get your team on-board to learn OpenAPI or worry about maintaining 10k line YAML files -- Optic takes care of all of that. Optic manages this process whether your development environment is running locally on your machine or out in the Cloud. No matter where you develop and test, you have confidence that every change to your API is reviewed, approved, and documented before it's released.