INFRA-107: nginx-reverse - Default target should be health check

Metadata

Source
INFRA-107
Type
Improvement
Priority
Major
Status
Closed
Resolution
Done
Assignee
Giovanni Tirloni
Reporter
Giovanni Tirloni
Created
2018-02-01T17:02:15.249-0500
Updated
2018-06-21T15:04:53.629-0400
Versions
N/A
Fixed Versions
N/A
Component
N/A

Description

When the load balancer is accessed directly without it being through a configured vhost, it defaults to the first server{}. When this server entry has some problem, the load balancer is reported as failed (which is a false positive, because the failed server is actually what's behind it).

Need to implement a default server{} that's a health check endpoint.

Comments

  • Giovanni Tirloni commented 2018-02-23T08:21:58.762-0500

    Here's an example of AChecker (the first backend) going down and both load balancers being reported as down too:

    The monitor IDRC - achecker.ca (https://achecker.ca) is currently DOWN (HTTP 502 - Bad Gateway).
    The monitor INFRA - load balancer i-0035 (https://i-0035.tor1.inclusivedesign.ca/) is currently DOWN (HTTP 502 - Bad Gateway).
    The monitor INFRA - load balancer i-0036 (https://i-0036.tor1.inclusivedesign.ca) is currently DOWN (HTTP 502 - Bad Gateway).

    All happened at 04:44pm and did not impact any other websites.

  • Giovanni Tirloni commented 2018-03-22T18:41:31.715-0400

    Implemented two new features in nginx-common:

    • Ability to define a default URL (anyone hitting the LBs without a Host header gets redirected there)
    • Returning OK 200 on a defined path (again, without Host header defined)

    UptimeRobot has been updated with the new endpoints for the load balancers.

    Sample output for default URL:

    $ curl -v http://i-0035.tor1.inclusivedesign.ca
* Rebuilt URL to: http://i-0035.tor1.inclusivedesign.ca/
*   Trying 205.211.169.48...
* TCP_NODELAY set
* Connected to i-0035.tor1.inclusivedesign.ca (205.211.169.48) port 80 (#0)
> GET / HTTP/1.1
> Host: i-0035.tor1.inclusivedesign.ca
> User-Agent: curl/7.55.1
> Accept: */*
> 
< HTTP/1.1 302 Moved Temporarily
< Server: nginx
< Date: Thu, 22 Mar 2018 22:24:58 GMT
< Content-Type: text/html
< Content-Length: 154
< Connection: keep-alive
< Location: https://idrc.ocadu.ca
< Strict-Transport-Security: max-age=0;
< 
<html>
<head><title>302 Found</title></head>
<body bgcolor="white">
<center><h1>302 Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host i-0035.tor1.inclusivedesign.ca left intact

    Sample output for health check:

    $ curl -v http://i-0035.tor1.inclusivedesign.ca/health
*   Trying 205.211.169.48...
* TCP_NODELAY set
* Connected to i-0035.tor1.inclusivedesign.ca (205.211.169.48) port 80 (#0)
> GET /health HTTP/1.1
> Host: i-0035.tor1.inclusivedesign.ca
> User-Agent: curl/7.55.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nginx
< Date: Thu, 22 Mar 2018 22:43:16 GMT
< Content-Type: application/octet-stream
< Content-Length: 0
< Connection: keep-alive
< Strict-Transport-Security: max-age=0;
< 
* Connection #0 to host i-0035.tor1.inclusivedesign.ca left intact