Hacking Apache's mod_proxy_http to enforce an SLA

Let's say you have a web-service exposing ReST APIs for your awesome data processor. 
It has data input/output APIs of various kinds. The software architecture consists 
of front-end apache servers and back-end tomcat plus various data stores. Apache's 
mod_proxy and some load balancer (HAProxy, mod_proxy_balancer) pushes the incoming 
requests to backend servers.

A client wants a guarantee that your APIs will accept requests and return valid data 
or response codes within XXms for 95% of requests (see Wikipedia's SLA for other 
examples of service guarantees). How can one be absolutely sure that the SLA is met? 
Now add in the wrinkle that there might be different SLAs for the various APIs. In 
addition, the SLA could specify that as close to 100% as possible of the requests
 return HTTP codes within the 2xx range.. suppressing any 3xx, 4xx or 5xx codes from 
coming back to the outside world.

The issues with making apache do this are as follows:

    * ProxyTimeout is global or scoped to the particular vhost
    * ErrorDocuments still return the error code (503, 404, etc)
    * No way to tie ErrorDocuments and ProxyTimeouts to particular requests.


A key insight from Ronald Park is to use mod_rewrite and then pass various environment
 arguments to mod_proxy that are specific to the URL being addressed by mod_rewrite. 
This was the approach taken by Ronald Park in his attempts to solve this problem in 
apache 2.0.x here and here.

The below example is a rewrite rule that makes no changes to the URL itself for a JS 
API presumably returning data in JSON.

RewriteRule ^/api/(.*).js\?*(.*)$ http://backendproxy/api/$1.js?$2 [P,QSA,E=proxy-timeout:900ms,E=error-suppress:true,E=server-pattern:Apache,E=allowed-statuses:2xx,E=error-headers:/errorapi.js.HTTP,E=error-document:/errorapi.js]

With the SLA enforcement modifications enabled, the URL will return data from the 
backend system within 900ms or a timeout occurs. At this point apache will stop 
waiting for the backend response and serve back the static files /errorapi.js.HTTP 
as HTTP headers and /errorapi.js as contents.

$cat /var/www/html/errorapi.js.HTTP
Status: 204
Content-type: application/javascript

$cat /var/www/html/errorapi.js
var xxx_api_data={data:[]}; /* ERR */

There are four environment variables the SLA hack looks for:

    * proxy-timeout: - time in seconds or milliseconds to wait until timing out
    * error-suppress: - true/false switch on suppressing all errors from the backend.
    * allowed-statuses: - list of allowed HTTP statuses to accept
                        - default is 2xx;3xx;4xx
    * server-pattern:  - a (sub)string that must be contained in Server field of the HTTP response
                       - implemented with strcasestr for substring match
    * error-headers: - file of syntax correct HTTP headers to return to the client
    * error-document: - file of content body to be returned to the client

Note on allowed-statuses directive:
  This can be a list of individual status codes like '200;301;404' etc.
  2xx means the entire class of 200 to 299 codes are allowed
  Default value of '2xx;3xx;4xx' allows all 200 to 499 codes. 
  Avoid spaces and commas in the pattern - NOT TESTED
  If the directive is left off the RewriteRule, then the default is used. 
  This directive is ignored if the error-suppress=false

Note on server-pattern:
  This is a substring to be contained in the 'Server' HTTP response header field.
  Example below
     - Server: Apache/2.2.3 (CentOS)
     - server-pattern:ABC will NOT match
     - server-pattern:Apache will match
  This is very usefull in a public Cloud like AWS where you might just be proxing off to an IP/Host not controlled by you!
  It is recommeneded that you update your back-end service to have a unique substring in its Server response header.
  Avoid spaces and commas in the pattern - NOT TESTED
  If the directive is left off the RewriteRule, then no check is done.
  This directive is ignored if the error-suppress=false
  
Leaving off the proxy-timeout will only suppress errors from the backend after the global 
timeout occurs. Leaving off error-suppress:true will ensure that the 5xx timeout error from
 mod_proxy_http is returned intact to the client. No other error supression is done.

There are multiple versions checked into github for Ubuntu 9.04's apache2 2.2.11 and Centos el5's.
It's advisable to diff the changes with the 'stock' file and likely re-do hack 
code in your version of apache 2.2. See Ron Park's code for 2.0.x and fold in the other mods 
supporting error-suppress etc.

The hack is has been tested in a production environment for almost 1 year.  Approximately 50B hits have passed through this code.

Patches were created like so:
diff -u mod_proxy_http.c.STOCK mod_proxy_http.c > mod_proxy_http.c.diff

Notes as of Jul 14, 2009:
- First version committed

Notes as of Jan 19, 2010:
- This mod has been in production for 6+ months and we love it.
- A code modification has been made to preserve 3xx (redirect) and 4xx (client errors) response codes.

Notes as of April 3, 2010:
- added a patch for newer version of http: httpd-2.2.3-31.2.el5_4_centos

Notes as of June 28, 2010:
- add parameterization of allowed status codes ; 2xx,3xx & 4xx are default 
- add Server response header pattern check

Notes as of November 3, 2010
- add millisecond capabilities to ProxyPass timeout directive
- add millisecond capabilities to ProxyTimeout directive

Notes as of January 14, 2011
- works unchanged on httpd-2.2.3-48 codebase