Document your organization's Vault capabilities with Optic
Optic helps teams write and maintain their first OpenAPI specifications. You don't need to get your team on-board to learn OpenAPI or worry about maintaining 10k line YAML files -- Optic takes care of all of that. Optic manages this process whether your development environment is running locally on your machine or out in the Cloud. No matter where you develop and test, you have confidence that every change to your API is reviewed, approved, and documented before it's released. Your documentation can provide details for your procedures, and assist in your governance goals.
Manage Secrets and Protect Sensitive Data with Vault#
At Optic, we want you to be able to safely and securely store and use your organization's secrets. There are many technologies available, from the cloud provider native solutions such as Azure Key Vault and AWS Key Management Service to open source solutions such as Hashicorp's Vault. Vault is an API-driven tool that secures, stores, and controls authorization for secrets. You can learn more about its claims and how it operates at Hashicorp's Vault project site. The important piece here is that where there's RESTful API driven tooling, there should be Optic. Optic helps me understand not only the APIS I develop, but the APIs I consume. The documentation generated by Optic demonstrates how I do my daily work. It's insight into the practices and procedures my team is currently using, and can help illustrate how we comply with policies such as Information Security.
A demo environment for Vault#
I set up a quick development environment for Vault to demonstrate how Optic can help document your secret store and make complying with security policies easier. I'll generally follow along with some examples provided in the Vault Getting Started walkthrough. You can follow along as well. The first step is installing Vault. There are many installation options, and I went with:
By default, Vault has a lot of options configured such as enabling TLS. Understanding how Vault is configured out of the box is important when deploying this in any production environment, as Vault is playing a critical role securing secrets. For our demonstration purposes, Vault provides a handy dev configuration which disables many of the recommended defaults. Secrets are not safe in dev mode, and I'd recommend against storing valid secrets without a stronger configuration. For our demonstration, though, this will work fine. We'll enable the dev mode with the -dev flag, and pass a root token in with -dev-root-token-id so it's consistent between runs:
That's was a valid unseal key when I captured it, though it's long been invalidated as it is generated fresh every time we start the dev mode server. Sealing/Unsealing is not necessary in this configuration, and we can safely ignore it. The only information we need is:
- our root token (
toor). - the Vault API address (
http://127.0.0.1:8200)
Configuring Optic to observe Vault traffic#
There's several ways to configure Optic, and in this case I'm going to choose to define my inbound and target URLs to match Vault's defaults:
I'll give my configuration a quick check to make sure it passes:
Since Vault is already running, the check passes. Hooray! I'll start Optic and we can start experimenting with Vault:
I'm almost ready to send requests through to the Vault service. It's API-driven, and I have many options such as Postman for building request libraries. In a deployed scenario, I'd recommend using a tool like Postman to build a repeatable test collection. For now, I'm going to use the Vault CLI and focus on Optic's documentation process. I'll need to tell Vault to go through Optic so we can observe the traffic with:
Storing the first secret#
Let's say I have an application that needs to be configured with a secret for some reason. Perhaps it needs to connect to another service. I'm currently injecting that secret into my environment somehow, and would rather fetch it from Vault. For a single secret, this may be overkill: you still need to authenticate with Vault in a deployed scenario. I'm forward-thinking, however, and know I'll have a lot of services with many secrets. I also know there are other features in Vault that we'll want to use, and I want to make building and maintaining my documentation part of that process from the start.
The first thing I recommend doing is checking that Vault has started successfully:
I cut out the details to focus on the important parts for my test. The dev vault is initialized and unsealed (I left version in as well for reference). If you're following along, you want your vault to respond with the same state. Now, let's store our first secret:
Oops, I forgot: I actually need two keys for this secret:
Did it work? Let's try fetching the secret we stored in secret/hello:
Success! I can retrieve just the excited key as well:
Optic observations#
Now I'll open up the Optic dashboard. My start output earlier reports it's available on http://localhost:34444/apis/1/diffs. When I open it up, I'll see quite a few items:
There's a few service endpoints fetched often, and some endpoints under /v1/sys (such as our status check) that we won't care about documenting. We'll ignore those later. For now, let's look into getting our secret data. I'll click through the GET /v1/secret/data/hello route. After documenting the route in the modals, I see that there's a lot going on under the hood! Optic observed all the traffic, and reduced it to the shape of the data on the right:
An important thing to note here: only the traffic shape is ever stored in the documentation. The traffic itself is never stored in the documentation. Raw traffic is observed and converted to its shape representation locally for documentation and comparison to any existing specification. We don't send your secrets off of your machine.
Going through the requests, Optic has detected our foo field is always present. It also notices that our excited field is seen in some requests, and confirms with us:
- the
excitedfield is allowed to be here, and not a mistake.
- the
excitedfield is an optional field, and its omission is not a mistake.
Now I'll commit the changes and accept Optic's default commit message. Optic will let me know that the endpoint is now documented, and the currently observed behavior does not differ from the documented behavior. That makes sense: we just documented all of the observations. I'll document the PUT route as well, and jump ahead to see our generated documentation:
Reducing noise: Adding ignoreRequest rules#
Optic configurations are stored in the optic.yml file in the root of your API project. One configuration option available is ignoreRequest. This allows you to define methods and paths to ignore. You can read more about how to ignore requests in our documentation. For now, I'm going to ignore all methods to the noisy routes we've seen. I'll stop my Optic proxy with CTRL + C to break out of the current session and update my configuration. By default, Optic includes an ignore rule for OPTIONS requests. When I'm done editing optic.yml with my editor of choice, my ignoreRequests section looks like:
Then, I can restart Optic:
Vault should still be running, and a quick invocation of vault status confirms this. I ran vault kv get secret/hello, and received my pair of keys as expected. Let's refresh the Optic dashboard at http://localhost:34444/apis/1/diffs:
Success! We can see two things here:
- There are no undocumented URLs visible. Vault is still making calls to service URLs, and I made a call to
/v1/sys/seal-status. The newignoreRequestsrules are working as designed. Optic passes through the requests and doesn't show them to us. Nice. - I requested my secret from
/v1/secret/data/hello, and Optic shows that there are no observed diffs. That means my request and response complies with the currently documented behavior of the API. When my application needs to fetch its secret, I know how to make that request successfully.
Observing hierarchy changes#
Let's say my application is ready to move off of my local machine and into my organization's infrastructure. My organization supports infrastructure under several apps across several environments, and expects a more structured topology as opposed to a flat list of applications. We also require different secrets for different environments. For example, I might need to have a structure that looks more like:
- staging
- helloapp
- prod
- helloapp
No worries! we can reorganize the layout with a few quick commands. Again, in an actual deployed environment, this process may be very different. I wouldn't recommend having an individual generate secrets via command line for everything, for example. However, we can use the Vault command line for demonstration:
To ensure coverage, our list of requests is growing. If we consider environment a parameter, we're also introducing some duplication of requests. This would be a great time to consider something like Postman Environments to parameterize your infrastructure environments, and collections to cover the capabilities you want to deliver with Vault. This will allow you to build repeatable API tests and validate compliance with Optic. For now, let's revisit the Optic dashboard:
Success! We see new undocumented routes for our latest endpoints. Going forward, as our infrastructure evolves and we change both the capabilities and the endpoints to deliver them with Vault, our documentation will stay up to date. When we see changes in behavior, such as new endpoints or new or optional fields, they will be called out by Optic. If there are no undocumented URLs and no endpoint diffs after testing our calls, we can be confident that our secret store is behaving as expected and our applications should be able to fetch their needed configurations in every validated environment.
Get started with Optic!#
You can start documenting your API today by setting up Optic with your project. We can help you integrate with your existing documentation tools during our office hours, and would be happy to take feedback on GitHub.