There are several standard filter types that correspond to the typical lifecycle of a request:

• PRE filters execute before routing to the origin. Examples include request authentication, choosing origin servers, and logging debug info.
• ROUTING filters handle routing the request to an origin. This is where the origin HTTP request is built and sent using Apache HttpClient or Netflix Ribbon.
• POST filters execute after the request has been routed to the origin. Examples include adding standard HTTP headers to the response, gathering statistics and metrics, and streaming the response from the origin to the client.
• ERROR filters execute when an error occurs during one of the other phases.

Alongside the default filter flow, Zuul allows us to create custom filter types and execute them explicitly. For example, Zuul has a STATIC type that generates a response within Zuul instead of forwarding the request to an origin.

How We Use Zuul

There are many ways in which Zuul helps us run the Netflix API and the overall Netflix streaming application. Here is a short list of some of the more common examples, and for some we will go into more detail below:

• Authentication
• Insights
• Stress Testing
• Canary Testing
• Dynamic Routing
• Load Shedding
• Security
• Static Response handling
• Multi-Region Resiliency

Insights

Zuul gives us a lot of insight into our systems, in part by making use of other Netflix OSS components.

Hystrix is used to wrap calls to our origins, which allows us to shed and prioritize traffic when issues occur.

Ribbon is our client for all outbound requests from Zuul, which provides detailed information into network performance and errors, as well as handles software load balancing for even load distribution.

Turbine aggregates fine-grained metrics in real-time so that we can quickly observe and react to problems.

Archaius handles configuration and gives the ability to dynamically change properties. 
 
Because Zuul can add, change, and compile filters at run-time, system behavior can be quickly altered. We add new routes, assign authorization access rules, and categorize routes all by adding or modifying filters. And when unexpected conditions arise, Zuul has the ability to quickly intercept requests so we can explore, workaround, or fix the problem. 
 
The dynamic filtering capability of Zuul allows us to find and isolate problems that would normally be difficult to locate among our large volume of requests. A filter can be written to route a specific customer or device to a separate API cluster for debugging. This technique was used when a new page from the website needed tuning. Performance problems, as well as unexplained errors were observed. It was difficult to debug the issues because the problems were only happening for a small set of customers. By isolating the traffic to a single instance, patterns and discrepancies in the requests could be seen in real time. Zuul has what we call a “SurgicalDebugFilter”. This is a special “pre” filter that will route a request to an isolated cluster if the patternMatches() criteria is true. Adding this filter to match for the new page allowed us to quickly identify and analyze the problem. Prior to using Zuul, Hadoop was being used to query through billions of logged requests to find the several thousand requests for the new page. We were able to reduce the problem to a search through a relatively small log file on a few servers and observe behavior in real time.
 
The following is an example of the SurgicalDebugFilter that is used to route matched requests to a debug cluster:

class SharpDebugFilter extends SurgicalDebugFilter {
   private static final Set<String> DEVICE_IDS = ["XXX", "YYY", "ZZZ"]
   @Override
   boolean patternMatches() {
       final RequestContext ctx = RequestContext.getCurrentContext()
       final String requestUri = ctx.getRequest().getRequestURI();
       final String contextUri = ctx.requestURI;
       String id = HTTPRequestUtils.getInstance().
           getValueFromRequestElements("deviceId");
       return DEVICE_IDS.contains(id);
  }
}

In addition to dynamically re-routing requests that match a specified criteria, we have an internal system, built on top of Zuul and Turbine, that allows us to display a real-time streaming log of all matching requests/responses across our entire cluster. This internal system allows us to quickly find patterns of anomalous behavior, or simply observe that some segment of traffic is behaving as expected, (by asking questions such as “how many PS3 API requests are coming from Sao Paolo”)?

Stress Testing

Gauging the performance and capacity limits of our systems is important for us to predict our EC2 instance demands, tune our autoscaling policies, and keep track of general performance trends as new features are added. An automated process that uses dynamic Archaius configurations within a Zuul filter steadily increases the traffic routed to a small cluster of origin servers. As the instances receive more traffic, their performance characteristics and capacity are measured. This informs us of how many EC2 instances will be needed to run at peak, whether our autoscaling policies need to be modified, and whether or not a particular build has the required performance characteristics to be pushed to production.

Multi-Region Resiliency

Zuul is central to our multi-region ELB resiliency project called Isthmus. As part of Isthmus, Zuul is used to bridge requests from the west coast cloud region to the east coast to help us have multi-region redundancy in our ELBs for our critical domains. Stay tuned for a tech blog post about our Isthmus initiative.

Zuul OSS

Today, we are open sourcing Zuul as a few different components:

zuul-core — A library containing a set of core features.

zuul-netflix — An extension library using many Netflix OSS components:

• Servo for insights, metrics, monitoring
• Hystrix for real time metrics with Turbine
• Eureka for instance discovery
• Ribbon for routing
• Archaius for real-time configuration
• Astyanax for and filter persistence in Cassandra

zuul-filters — Filters to work with zuul-core and zuul-netflix libraries

zuul-webapp-simple — A simple example of a web application built on zuul-core including a few basic filters

zuul-netflix-webapp — A web application putting zuul-core, zuul-netflix, and zuul-filters together.

Putting everything together, we are also providing a web application built on zuul-core and zuul-netflix. The application also provides many helpful filters for things such as:

• Weighted load balancing to balance a percentage of load to a certain server or cluster for capacity testing
• Request debugging
• Routing filters for Apache HttpClient and Netflix Ribbon
• Statistics collecting

We hope that this project will be useful for your application and will demonstrate the strength of our open source projects when using Zuul as a glue across them, and encourage you to contribute to Zuul to make it even better. Also, if this type of technology is as exciting to you as it is to us, please see current openings on our team: jobs

• Mikey Cohen — API Platform
• Matt Hawthorne — API Platform

See Also: