I'm sure I'm not the only one who got burned by this during the ARRL Sweepstakes - periodically the RBN Telnet server would go down, throwing everyone off, reject connection requests for a few minutes, and then come back on - but unless your client had automatic re-connect capabilities, you might not notice until the spots stopped flowing, and then, if you tried to reconnect, you might or might not get back on..
We (mainly Felipe and Nick, F5VIH) have been studying the logs and configuration details for a few days, trying to figure out what is going on.The spot volume doesn't appear to have been to blame - only 470k spots for the busiest 24-hour day, or about 5.5 per second on average ("only", he says!). Instead, it seems to have been the number of users - we're once again victims of our own success.
There are several potential avenues for fixing this - the best would probably be to adjust the current server's parameters to increase its inbound bandwidth and other capacities very substantially, but that may not be feasible with a Virtual Private Server such as we're currently using. We could get another server, and set up a round-robin DNS so that connection requests are routed to them alternatively. Fine, except that this requires the added server to have a fixed IP address, and if either server gets overburdened and crashes, then half the users would find themselves getting rejected or thrown off until it recovers. A third possibility might be to move the whole thing to another server that we don't share with anyone, and that we can set up for high volume. A fourth, crude solution would be to set up another server with an entirely separate URL, and publicize its availability in the hope that the users themselves will redistribute among the servers.
Anyhow, we're scrambling to figure it out and solve the problem before CQWW CW in ten days' time - please stay tuned.
73, Pete N4ZR