Random Quick Zoning Disconnects

Thanks everyone for providing additional information. I have been investigating this for days and I have a hopeful server code fix that may resolve this.

 

I hotfix to include this fix. So we'll see how it goes.

 

It always seems so simple once you understand the problem and have already solved it, but this was an interesting puzzle.

BLUF: The reported behavior was triggered by packet reordering in the network. I fixed it by suppressing an additional packet that was breaking the protocol when it arrived out-of-order and that packet wasn't needed in the first place.

The zone failure problem resisted simple debugging by swapping out components like the server, the client, or the user because of the combination of factors required to produce it. There is a sensitive portion of the protocol where the client and server are exchanging control messages to gracefully disconnect from each other that failed to process the disconnect on the client (leading to a timeout) only when two specific packets sent from the server arrived out of order at the client.

Investigation:
Users report that switching ISPs or using VPNs can either make the problem go away or make it start happening. Normally the simple conclusion would be that there is something different with the user's equipment/software that is disturbing the 'default' and that's why it doesn't work, possibly DNS errors, and the user should just figure it out, but this problem is reported by multiple users using both the Macintosh and Windows clients, with or without add ons, and they're using all different equipment/software for their network so that's not a good explanation.

One of the common failure modes I considered is router state filling up - a home router doing NAT can only hold so many connections at a time and people resolve failures by rebooting it which temporarily makes it forget all connections and it works for a little while until it's 'full' again. That doesn't apply here because while it's a common failure, the reported issue is too specific in how it presents and it's affecting too many users to explain it that way.

The EQ network protocol uses a reliable transport that resends packets until acknowledged, and there is some timing sensitivity here, so I considered that high latency is somehow aggravating a race condition problem. High latency can happen even for users close to the server due to an issue called bufferbloat, and this seemed like a promising lead. I attempted to simulate high latency and varying degrees of packet loss using linux qdiscs and ifb but the transport protocol's resend logic made it work reliably regardless of what I tried - asymmetric latency, high latency, varying latency.. and even though I didn't find the problem, it's nice to see that it's working so well and able to tolerate poor quality connections.

Rimidal/showstring and benoeb used a VPN to fix the issue while playing since their native connection was experiencing the problem. This likely increases latency further so that's another ding against the latency trigger idea. These VPN services exist primarily for watching out-of-region Netflix shows so they offer multiple servers labeled by country that hopefully geoIP to the where the user wants to be, and after trying various servers in the list we found that some of them produce the zoning issue and some don't. Then I considered maybe there's some firewall or other device that filters/rate limits/mangles packets as this is something that happens with UDP as network admins try to mitigate abuse and then forget to remove the filters afterward. It can go unnoticed because HTTPS is TCP and continues to work even with UDP filtered. This theory kind of falls apart because the failure is so specific/predicatble where it happens and it only happens during zoning - still, it is possible that a certain packet involved in zoning matches some IDS signature and gets mangled/filtered so at this point I had to go for capturing packets. Looking at EQ packets in tcpdump/wireshark and counting bytes is something I haven't had to do since I started doing eqemu 25 or so years ago while figuring out the basics, so I was really hoping to find the problem some other way.

I ended up signing up for the VPN service that showstring is using so that I could try to reproduce the problem myself, and I needed to do it on a dev server not just on the live one, so that I could experiment. At this point showstring had only been able to produce the failure on the TAKP server and not TLP, p99 or my dev server but without being able to reproduce and try things I still had nothing to go on. I ended up building out several dev server copies of TAKP and eventually finding a VPN server + EQ server combination that did produce the problem which was the key to me being able to actually move forward. So I thought that I would just capture on both sides and try to compare, see if any loss happened, or packets got changed in transit. After accounting for clock differences and latency to align the client and server captures, I hashed every packet and compared the list to see that every packet sent at the server was actually seen on the client and vice versa, and it came up clean even though it was producing the zone failure. So that's a good sign kind of, it's not some weird mangling issue that's affecting people all over the internet on different ISPs, it's probably something we can fix then. What I did not do is compare the order of the packets at this point - it's a large data set, so I just went with the idea of verifying the contents matched, that every packet hash was present in both lists. I did not find any packet mangling issues so I went back to focusing on the client's code to try to understand where it was failing and what circumstances could trigger that.

I have some instrumentation in my eqclientmod (on github) that logs debug messages in the client and I had received some of those logs and packet captures from showstring so I was able to confirm that my repro was correct and hitting the same failure mode. I discovered several secondary issues with the zoning process, things that could be improved, but turned out not to be the cause of this failure, so then I went lower level and started studying the EQ protocol headers themselves. There are combinations of bits that are used to signal for flow control, and I was able to decompile and compare the signalling between the client and server. I knew the failure was that it was hanging while trying to disconnect waiting for the finalization packet, but I could see the finalization packet in the packet capture so why wasn't it processing? It turned out there were 2 of those packets that got sent back to back. In our code, it's just something like Disconnect() that ended up getting called twice during the zoning process, and these are reliable messages meaning they have a sequence number so they get retransmitted until acknowledged. When I saw the 2 close messages and the sequence numbers on them, I realized that the client received the one with the higher sequence number first and the lower sequence number second. Studying the client code, when this happens it ends up ignoring the lower sequenced one, so the protocol state desyncs and the server keeps trying to retramisit while the client ignores it and eventually times out. This only happens some of the time, when the packet order is swapped by the network, which is not ideal but it is a thing that happens at this low level. TCP is tolerant of this, as long as it's not a high rate of reordered packets, so facebook loads, youtube works, nobody knows it happened. It would be better if the network didn't do this, but it does, and applications need to tolerate it. EverQuest does tolerate it well, but this particular packet is handled in a special way and it is improper for the server to send two close packets anyway, but it's been mostly working because this is a random occurrence.

It's an error to send multiple finalization packets, so I fixed it by making the server only send one.