Update on server issues

Discussion in 'General Discussion' started by Callixtus, May 11, 2015.

  1. robregen

    robregen Administrator Staff Member

    Messages:
    8,390
    another thing that you guys can do, is log on the dev server to play on there like you would on takp to see if there is any differences.
     
  2. Elroz

    Elroz I Feel Loved Staff Member

    Messages:
    2,286
    the massive lag happens on the dev server when you have a corpse as well. Zoned to pok without lag, went and died. Loaded into pok with a huge spike, zoned to qeynos with a huge spike, zoned back to pok with a huge spike, back to qeynos with a spike / desync, loaded in from desync with a huge spike, looted corpse and zoned back to pok with no lag. This probably rules out my takp magelo idea unless it's connected to dev server as well. Also I notice when looting my corpse the ping jumps up every time I loot an item, and it goes way up when I loot a bag full of items.
     
    Ghakim likes this.
  3. Quaos

    Quaos New Member

    Messages:
    153
    I can also log on a box (box b) while box a is already logged on and get pings from 300-700 that spike up to 20-25K on my box a.
    I get the same lag spikes when someone else logs on then both my boxes get the same spikes.
     
  4. Mascha

    Mascha People Like Me

    Messages:
    553
    Just had an interesting experience in POK. 1 character (enchanter) was in POK, seemed normal. Other character (druid) went LD porting into POK. Enchanter watched POK port in, did not see the LD character, waited ~15 minutes. The LD druid coudn't log back in, once entering the world the screen went black and went to server select after 5 minutes or so. Enchanter in POK didn't see the druid, but suddenly, after 2 login attempts of the druid, enchanter sees the druid suddenly appear at the port in point (visually for the enchanter - the druid log in screen remained black).

    After another 5 minutes or so the druid's screen went to server select, the druid disappeared for the enchanter watching in POK, and the druid could log in the next attempt.

    To me it looks like an LD character is not only LD while the character can be seen by other characters which are on, but the server seems to think the character is alive and online. The fact that the character disappears for a while and then appears (while still LD, looking at a black screen) looks to me like the server is switching between LD status and Online status. Could that be part of the problem?
     
  5. Mascha

    Mascha People Like Me

    Messages:
    553
    Another riddle. Enchanter is online in POK, sees druid. Druid sees Enchanter, they can trade. Druid does not see necro. Enchanter sees necro, but cannot trade. Necro sees druid and enchanter, but cannot trade. Logging necro off, necro's game instance crashes with Windows error message.
     
  6. Mascha

    Mascha People Like Me

    Messages:
    553
    And, another observation, but perhaps interesting for brainstorming. The character login screen behaves differently for different kind of crashes. Sometimes when you log back in after a crash, the character you used last is highlighted in the character list. Sometimes the uppermost character is active (marked), even it was not the last played. This means to me that the game could not log off correctly - I think that if the game closes correctly, it remembers which character was played last and marks it upon re-logging. If not (and the uppermost character is marked), the game could not close correctly.
     
  7. Mascha

    Mascha People Like Me

    Messages:
    553
    And the last one for today: Druid and Enchanter were online in POK, waiting for necro to log in. Necro logs in, Enchanter and druid can see him, druid targets him to buff and suddenly necro disappears, the necro disappears in target. /tar xxx does not work. Necro is on druid's tracking list. Necro sees both, but cannot trade with either.

    If druid types /assist necro while necro targets enchanter, druid gets enchanter as target. That means that the server recognizes the necro, but druid and enchanter cannot see or target necro.

    A /who on all three shows 7 players (all the same players). Just confirmed with other players: They cant see necro, but can assist him.

    Necro can bank and sell to vendor, but cant use Jboots.
     
    Last edited: May 12, 2015
  8. Haynar

    Haynar Administrator

    Messages:
    3,637
    I think the data spike is causing clients buffers to fill up or toggle LD. So spawn entry data never makes it to some if the other clients in the zone. Making a desync condition. Not the cause, but a side effect that we could probably compensate for.

    H
     
  9. robregen

    robregen Administrator Staff Member

    Messages:
    8,390
    okay Looks like we found the issue. There was a script that would run a query and check for items on corpses anytime someone zone in. We removed that part of the scripts. and looks like the issue has been resolves.
     
    Sketchy, Pinoit, Mokli and 2 others like this.
  10. Ghakim

    Ghakim Well-Known Member

    Messages:
    169
    YAYAY thank you Robregen and the rest of the dev team!!!

    Awesome!!
     
  11. Pinoit

    Pinoit Well-Known Member

    Messages:
    294
    Wow nice bug find! Talk about obscure...
     
  12. Callixtus

    Callixtus New Member

    Messages:
    58
    HUZZAH!!!

    Great news Robregen!

    Thanks are in order to you and the team. Some might not always remember that you guys are doing this in your spare time, so thanks again!

    Callixtus/Iomadan/Xalithix/Nilno
     
  13. hrududu

    hrududu New Member

    Messages:
    7
    Nice catch!
    Makes me wonder though: why would there be a need for a script like that? And what breaks by removing it?
    Time will tell I guess. For now, it's great news!
     
  14. robregen

    robregen Administrator Staff Member

    Messages:
    8,390
    That script was part of a check inventory including corpses plugins that was in with PEQ, that has been part of the script for a long time. I recently wrote a script for Plane of Sky that uses that plugin. Unknowingly, that plugin had devasting side effect serverwide. Anyhow, that corpse check part of the script is gone and it is not needed.
     
    Last edited: May 13, 2015
  15. Elrontaur

    Elrontaur People Like Me

    Messages:
    1,291
    NUKE IT :mad:
     
  16. Bum

    Bum I Feel Loved

    Messages:
    2,647
    Sounds like Bumm broke the server
     
    Rdyott likes this.
  17. Neealana

    Neealana People Like Me

    Messages:
    1,261
    The moral of this story: LOOT your corpses or YEA you will bring down a PLAGUE upon THIS world and the MIGHTY shall look DOWN upon us and see us SUFFER and succor us NOT.
     
    Speedz likes this.
  18. Sturmm

    Sturmm Well-Known Member

    Messages:
    317
    Don't forget to count to three before thou throw the holy hand grenade of antioch. Two is to little, and four is too much. Five is entirely out of the question.
     
  19. Mascha

    Mascha People Like Me

    Messages:
    553
    Has the patch already been implemented? I am looking forward to raid tonight if everything works fine :)
     
  20. Haynar

    Haynar Administrator

    Messages:
    3,637
    My goal is to take this bug, and use it to simulate hitting a lag/packet loss situation. And see if I can make the clients survive it without getting disconnected. Including the desync issue where chars cant see each other.

    I am going to have fun with this one.

    Haynar
     
    Rdyott, robregen and Speedz like this.
  21. Dranthor

    Dranthor Active Member

    Messages:
    162
    Awesome. If I could raid without DCs every 5 minutes, I'd be a very happy cleric.
     
  22. Speedz

    Speedz Administrator Staff Member

    Messages:
    2,469
    You should have seen what we went through to find this. Was nutty.
    Also, good idea Haynar, we finally have a way to artificially induce massive packet loss and server hangs to hard test stability :)
     
  23. Haynar

    Haynar Administrator

    Messages:
    3,637
    Once you can simulate the condition, I am really good at figuring out solutions. I already have ideas. And I have dug through this netcode quite extensively before.

    H
     
  24. robregen

    robregen Administrator Staff Member

    Messages:
    8,390
    We are on the current patch now and back on track. Enjoy! ;)
     
    Neealana and Ghakim like this.
  25. Quaos

    Quaos New Member

    Messages:
    153
    You guys are awesome!
     
  26. PompeiiEQ

    PompeiiEQ Active Member

    Messages:
    83
    Big thumbs up. I'm off all day. Just got finished my errands. And am excited to play all day today :)
     
    robregen likes this.