Running Optic in your CircleCI CI/CD environment

Running Optic in your CircleCI environment assures that differences from documented behavior or undocumented routes in your tests can be caught before a pull request is merged. Optic lets your CI pipeline know that the intended behavior of your API is documented, and alerts you to unintended behavior before it gets to deployment. If you don't use CircleCI, let us know your CI tool and we'll be happy to help.

Running Tests in CircleCI with Optic#

Before you can run your tests in CircleCI, you'll want to make sure your tests are running normally locally. Even if you don't run tests every time, or if you run your tests elsewhere, there will be some setup required to get your tests running. You'll also have to run your tests through Optic. Not only will your tests run, they'll be observed by Optic and compared to your documentation. Then, you can use Optic in CircleCI to run these same observations throughout your integration and deployment journey.

Integrating Tests using Optic#

There's a few tasks required to integrate your tests with Optic into CircleCI:

  • Check out your project
  • Install Optic and any dependencies
  • Run your tests
    • If they pass, assure the shell exits with exit code 0.
    • If any tests fail, assure the shell exits with exit code 1.

The following CircleCI config file accomplishes this for your example project. Note, that this is only a configuration file. It defines the actions CircleCI will take. CircleCI must be integrated with your central repository, such as GitHub, before it will take any actions. The --exit-on-diff flag will result in an exit code of 0 (normal) when no diffs are detected between the test traffic and the specification. On any difference in behavior, the program will exit with exit code 1, signaling a failure to CircleCI.

When this is committed to the .circleci/config.yml directory and pushed to your central repository that's integrated with CircleCI, the optic-ci job will run on the appropriate triggers. The configuration also includes the node/test job, as this is a node project. It's not necessary, though it provides a nice extra check that your project can build and basic tests pass on their own (if present).

version: 2.1
node: circleci/node@3.0.0
- node/test
- optic-ci
executor: node/default
- optic-ci
- checkout
- run:
name: Run Optic CI
command: |-
npm install @useoptic/cli
npx api run test-cci --collect-coverage --exit-on-diff

Catching Undocumented Behavior#

Your repository is now set up to run your tests automatically through Optic using CircleCI. Let's say you've added some new features to your project. This includes some new API endpoints. Your team likes the principles of test-driven development, and as part of your new work you've also written tests for the intended behavior. You've run the tests locally, and confirmed the behavior manually, forgetting to run the tests through Optic. Your new features are working, but are currently undocumented. The next time your code runs through CircleCI, your tests will run through Optic. The new routes (or diffs from existing documented behavior) are caught by Optic, which reports "Unexpected API Behavior". The process exits with exit code 1:

API changes will fail this test

This bubbles up to your central repository, where the check reports failure. Whoops! Now, anyone reviewing your work (such as in a pull request) can see that the Optic check has failed. Even if the tests all pass, if there's any difference in behavior or undocumented routes, Optic will let you know. Your team lead is really nice, and helpfully points out that the deliverable isn't ready until it's documented.

Check failure reported in the pull request

Documenting the Behavior#

The good news is, documenting an API is quick with Optic. Run the tests through Optic to generate traffic, check the dashboard, and update what's missing. The new documentation is committed to the repository and pushed to your central repository. This time, all endpoints are documented and the test traffic doesn't differ from the documentation. Success!

Test passes

Your team lead reviews your work. They're pretty confident that the code looks right, and can back that up with the successful test runs. They also know that the documentation is up to date, and that not only have the tests passed but the traffic matches the documented behavior. They can also pull the branch and read the documentation themselves if they want to dig deeper.

The successful test result is reported in the pull request

Adapting Optic to Run in Your Environment#

We'd be happy to add more examples based on your CI pipeline. We understand that we can't cover every environment, and are happy to chat with you more if you run into any issues. Please schedule some time to chat with us, or reach out to us via GitHub Issues for more information.