An interesting issue arose when trying to configure two L5 rules behind a SSL proxy on a Cisco CSS 11500.
When doing SSL termination on the CSS load-balancer, a ssl-proxy-list is configured to add a virtual server that ties the SSL VIP to the plain-text HTTP VIP used by the proxy.
Read up on configuring SSL termination on the CSS 11500 if you’re not familiar.
Normally, a SSL rule VIP is proxied to a single, matching plain-text HTTP VIP when you need to ensure the site is protected by SSL. This is done with two L4 rules, one matching [port] :443 and the other on :80. It’s not a requirement that the two VIPs match, but doing so will make your config easier to understand and conserve IP space. See the post on *todo* for CSS HTTP to HTTPS redirection.
In a special case, L5 rules were needed for the SSL proxy so traffic could be split on the URL to two different web farms. One L5 rule would match url “/*” and the other url “/something-more-specific/*”.
Upon testing, tcp connections stuck to one L5 rule or the other regardless of the request URL and appeared completely random at first until further testing showed the pattern.
Established TCP connections were not getting rebalanced (or remapped) as they normally do even with rule “persitence” enabled and the global setting of “Persistence Reset Method: Remap”. On a sidenote, initial TCP connections were delayed for 3 seconds. This became relevant later.
Searching the web turned up a couple of Cisco supportforms articles — see the references below — that first discussed disabling the rule persistence which was earlier confirmed to not matter, but that article lead me to another topic that described the same symptoms and hit on the root cause of the issue. The ssl virtual server config with highlighted line triggering issue follows.
ssl-server 10 vip address 10.x.y.z
ssl-server 10 cipher rsa-with-rc4-128-sha 10.x.y.z 80
ssl-server 10 cipher rsa-with-3des-ede-cbc-sha 10.x.y.z 80
ssl-server 10 cipher rsa-with-rc4-128-md5 10.x.y.z 80
ssl-server 10 rsakey cssrsakey2048
ssl-server 10 rsacert star-example-20131017
ssl-server 10 http-header static “X-Forwarded-Proto: HTTPS”
ssl-server 10 http-header insert-per-request
A ssl-proxy-list containing a cipher of rsa-with-3des-ede-cbc-sha leads to the failure of L5 matching. Any CBC (Cipher-Block-Chaining) cipher fragments the HTTP GET request, making the CSS unable to reassemble the packets for L5 matching. Now it might be possible to have set the global setting of “tcp-ip-fragment enabled” so the CSS could reassemble, but that config was not tested as it would have a potential impact to every other production rule. The workaround was to simply remove the CBC cipher in the ssl-proxy-list virtual server with “no ssl-server 10 cipher rsa-with-3des-ede-cbc-sha”.
Regarding the 3-second delay on SSL proxied connections, Cisco documented this CBC bug in CSCtg20158:
When you configure the CSS for SSL termination with HTTP header insertion and a clear-text Layer 5 content rule on the backend, there may be a 200-millisecond (ms) delay making the connection to the backend server. One SSL packet on the front end may become multiple TCP packets on the backend (clear text rule) when the HTTP header is inserted. The SSL module is acting as the client to the SP and the 200-ms delay is expected because the SP waits 200 milliseconds to send the TCP ACK for the second TCP packet. The CSS detects that the client is actually the SSL module and skip the 200-ms timer on the SP.