A Brocade ADX Real server is identified by it’s IP address and a logical name. If the server is in a subnet that is local (directly connected) on a subnet that the ADX is directly connected to (same broadcast domain), that server is defined as a real server. It is possible to configure a real server that is NOT in a subnet that is directly connected. We call these servers “remote servers”. Essentially, a real server and a remote server are the same thing: A logical representation of a server using an IP address and a logical name. The difference between these two types of real servers are the manner in which the ADX performs health checks. See the health checks section for additional information.
Real servers are defined with the command: “server real” , while remote servers are defined with the command “server remote“. From this point forward, we will refer to the servers as “real server” but note that this information also applies to “remote servers”. If you have a Brocade ADX in your Office and need assistance configuring this device, please feel free to contact me and I can help configure this device.
Real Server Ports
Ports are defined on the real servers. These are the ports on which the actual server listens. For example, if a back-end server has an Apache or IIS instance which is listening on 80 and 443, then the real server will need to have ports 80 and 443 defined if we want the ADX to send traffic to the back-end server on these ports. We still need to bind (assign) these real servers and real server ports to a virtual server (VS) before the ADX will send any traffic to them. Note that the command port http url “HEAD /” is a default health-check setting that automatically shows up when you define an http port. The port ‘keepalive’ command is necessary if you wish for the device to constantly check the life of the server. Without this command, the ADX will only check the devices port health once and will not check again.
server real WEB1-192.168.60.39 192.168.60.39 port http port http keepalive port http url "HEAD/" port ssl port ssl keepalive
Remote Server Ports
By default, a virtual server will not send traffic to a REMOTE server unless ALL of the REAL server ports are down for a particular VS. That is, if I have 3 servers on port http; 2 real servers and 1 remote server, ONLY the 2 REAL servers will receive traffic. If BOTH real servers on port http fail health checks, then and only then, will the VIP send traffic to the remote server. This is the default behavior. This behavior can be over-ridden with the virtual server port command lb-pri-servers. Once the lb-pri-servers command has been added for a particular port in the virtual server configuration, remote servers will be added to the VS rotation the same as if they were local real servers.
server virtual VS_18.104.22.168 192.168.59.34 port http port http lb-pri-servers port ssl port ssl lb-pri-servers
In order to use a non-standard port like 8180 on a real server, we need to tell the ADX what kind of port this is. (TCP vs UDP for example). To do this, we create the port profile from global config:
Non-Standard Ports on Real Servers
server port 8081 session-sync tcp
- server port 8180 = Create port 8180
- session-sync = On an HA pair, we need to keep the TCP session data consistent so that in the event of a failover, the TCP sessions will continue to operate without needing to rebuild a new session.
- tcp = configures this as a TCP port 8180
Verifying Real Servers
Once configured, we can see the real-time status of these ports on the ADX with the “show server real” command.
A port that is bound to a virtual server and passing health checks will have a status of ACT (active) . If a real server is failing health checks , the status will be FAL (failed) or TST (testing) . If the port has been administratively disabled , it will be marked DIS (disabled) . If the real server port has been configured on the real server but it has not been assigned (bound) to any virtual server, it will show UNB (unbind) . Note that there are NO health checks sent to a real server in the unbound state.
You will also notice in this show server real output that there is a default port configured on the real server but this is not reflected in the config output of the real server. This is a range of ports that the real server could potentialy listen on. DO NOT BIND THE DEFAULT PORT to a virtual server. Default ports do NOT use health checks. As such, a down server could continue to be assigned traffic by the load balancer and will result in failed connections from clients. Again, the default port should NOT be used or bound (assigned) to a virtual server and you can ignore the information in the show server bind output.
vlb1-adx1000#sh server real WEB1-192.168.60.39 Real Servers Info ======================== State(St) - ACT:active, ENB:enabled, FAL:failed, TST:test, DIS:disabled, UNK:unknown, UNB:unbind, AWU:await-unbind, AWD:await-delete HLD:held-downName: WEB1-192.168.60.39 State: Active Cost: 0 IP:192.168.60.39: 1 Mac: 0050.56b7.5ea1 Weight: 1/1 MaxConn: n/a SrcNAT: not-cfg, not-op DstNAT: not-cfg, not-op Serv-Rsts: 0 Rx throughput: 0 Kbps Tx throughput: 0 Kbps tcp conn rate:udp conn rate = 0:0, max tcp conn rate:max udp conn rate = 1:0 BP max local conn configured No: 0 0 0 0 0 0 BP max conn percentage configured No: 0 0 0 0 0 0 Use local conn : No SIP current TCP connections = 0 Port St Ms CurConn TotConn Rx-pkts Tx-pkts Rx-octet Tx-octet Reas ---- -- -- ------- ------- ------- ------- -------- -------- ---- default UNB 0 0 0 0 0 0 0 0 http ACT 6 0 1 9 10 624 684 0 ssl UNB 0 0 0 0 0 0 0 0 Server Total 0 1 9 10 624 684 0
You can see what virtual servers (VS) to which the real server is bound by issuing the command “show server bind“.
vlb1-adx1000#sh server bind Bind info Virtual server: VS-22.214.171.124 Status: enabled IP: 192.168.59.34 symmetric VIP state: Owner http -------> WEB2-192.168.60.38: 192.168.60.38, http (Active) WEB1-192.168.60.39: 192.168.60.39, http (Active)
Troubleshooting Real Servers
See the Articles on Real Servers Health Checks(Link soon) and Troubleshooting Health Checks(Link soon) to understand why a server may be marked DOWN or FAILED.
Take a Real Server out of Rotation (Disable a Real Server)
By a Health Check
A server can be taken out of rotation by a customer by modifying a health check page. See the L7 Content Check(Link soon) in the Health Checks Article.
Administratively Down the Entire Real Server on All Ports
You can down an entire server on all ports:
server real WEB1-192.168.60.38 port disable-all
Administratively Down the Individual Ports on a Real Server
server real WEB1-192.168.60.38 port http disable
To reverse these actions, simply “no” them out:
server real WEB1-192.168.60.38 no port disable-all server real WEB1-192.168.60.38 no port http disable
(DEPRECATED)NOTE: If you are attempting to disable an alias port , as of version 12.400g and below, you must disable the actual alias port. Simply disabling the real-port is not sufficient. Future versions are supposed to fix this so that use-master-state is honored completely.
Deleting a Real Server
By default, the real servers will NOT unbind if there has traffic flowing through them. It is VERY important that you make sure the following global command is in place BEFORE you attempt to delete a real server:
If you fail to do this, the real servers may get caught in a “Awaiting Unbinding State” (AWU). Issuing the server force-delete command AFTER you have attempted to delete a real server is ineffective. Due to persistent connections from clients, this state will last a long time and will not allow you to rebind the same server until this clears. You can clear this state by performing the following:
clear server all-session real-server WEB1-192.168.60.39 port http
See the GracefulShutdownofRealServers for more details.
Best Practices for Deleting a Real Server
- IMPORTANT! Verify that server force-delete is in the running configuration
- Disable All Ports on the Real Server
server real WEB1-192.168.60.38 port disable-all
- Make sure all connections have terminated to this real server This can be up to 2 Minutes with “server force-delete” enabled. Without server force-delete enabled, the connections could remain indefinitely
show server real WEB1-192.168.60.38
- Unbind the real server from the virtual server
server virtual VS-126.96.36.199 no bind http WEB1-192.168.60.38 http
- Delete the Real Server
no server real WEB1-192.168.60.38
Real Servers is the Brocades mechanism for referencing the Servers used by the VIP. These are required, and will be referenced, within Virtual Servers on the Brocade ADX. The Virtual Server will make a loadbalancing decision and forward traffic to one of these servers.
Graceful Shutdown of Real Servers
By default, if an application on a real server becomes unavailable (L4-L7 Health Check fails), but the real server itself is still up and responding to L3 ping health-checks. The ServerIron ADX continues to forward existing connections to the real server, but new connections will not be sent to the server. There is an exception to this rule if you have reset-on-port-fail enabled.
As a result of the above behavior, when administratively disabling a real server or real server port, the ADX will wait for all existing connections to terminate, HOWEVER LONG THAT TAKES. If a client is continuously updating the connection, the server may NEVER close that connection. So, the default behavior of suspending a port on a real server is a graceful shutdown (“Bleeding”) of connections.
A tech may decide to blindly delete a real server. This process is subject to the same rule. The ADX will wait for all connections to terminate, HOWEVER LONG THAT TAKES. This often has an unintended side effect of placing the server into an AWU status (Awaiting unbinding). As a result, the real server may not be re-bound to a VS until all connections have terminated. Often, persistent clients keep this real server in an indefinite AWU status. To clear this state, you will need to run the following command.
clear server all-session real WEB1-192.168.60.38
See the section Best Practices for Deleting a Real Server before you ever attempt to delete a real server.
The global command “server force-delete” should always be used when using a Brocade. There a few cases where this may be good for your company to not use it but it is best to always have this enabled. The server force-delete command ensures that existing sessions are terminated within two minutes. In other words, it’s a graceful shutdown unless a client holds a connection open for longer than 2 minutes. After 2 minutes, the ADX will force close the connection. In code 12.3.01 and older, the session may last up to 5 minutes.
NOTE: The global configuration for “server force-delete” MUST be entered BEFORE attempting to delete or shut down a service. Adding the command AFTER attempting to delete or shut down a real server has no effect.
The command “Reset-On-Port-Fail” will reset existing connections when a real server port becomes unavailable either by being administratively disabled or by failing a health check. On the ADX, disabling a server normally allows for graceful termination of connections, however, “reset-on-port-fail” will override graceful termination. This is by design. The command may have been better written as “reset-on-port-down” because the logic on the ADX doesn’t discriminate between administratively-down (disabled) and down (failed), it simply sees the port as down and forces all connections off of the real server.
If the desired outcome is a graceful shutdown of a real server port, before disabling a port care must be taken to ensure that reset-on-port-fail is not enabled on the real server port, else the shutdown will be forced and not graceful.
What About Failing Health Checks
When the real server stops responding to health checks, the ServerIron ADX removes the server from the VIP. That means no NEW connections will be sent to the real server. However, existing connections will still be sent to the real sever upon a health check failure. Again, if server force-delete is enabled, the connections will have up to 2 minutes to terminate.
There are a few exceptions to this:
- If the server no longer resolves ARP requests, a log entry indicating Mac Address Deletion will be written and the session will be rebalanced immediately.
- If the server no longer responds to L3 ping health checks, the session will be rebalanced immediately.
- If the real server PORT has reset-on-port-fail configured, the existing connections will be reset so that the client can make a new connection and be rebalanced.
- If the VS is using a layer 7 cookie for server persistence and the server fails a health check, the session will be rebalanced immediately.
Please feel free to reach out to me via my contacts page if you have any requests on more content or if you have any questions. If you want any help with an ADX in your Office, please feel free to contact me as well and I will work with you on configuring, upgrading or migrating the ADX.