Update on server issues

Discussion in 'General Discussion' started by Callixtus, May 11, 2015.

  1. Callixtus

    Callixtus New Member

    Messages:
    58
    Afternoon all,

    Is there any update on the server stability issues?

    Thanks!

    Callixtus/Iomadan/Xalithix/Nilno
     
  2. robregen

    robregen Administrator Staff Member

    Messages:
    8,386
    Well, we had test this locally and on dev. We rolled back and the problem persisted. We don't know if this a network issue, code, database or a combination of the two or all. There will be no more patch until this issue gets resolve. If we do get a patch, this will contain a possible fixed. This has been a nightmare for us and I'm very sorry for the inconvenience that it has cause for you players.
     
    Elrontaur likes this.
  3. Callixtus

    Callixtus New Member

    Messages:
    58
    Thanks for the update Robregen!!!

    And speaking for myself, no apologies needed whatsoever.

    Cheers!

    Callixtus/Iomadan/Xalithix/Nilno
     
    Cutty likes this.
  4. Sturmm

    Sturmm Well-Known Member

    Messages:
    316
    Sheet happens Rob, no worries. Thanks for working so hard to try and resolve it, we appreciate it
     
  5. Haynar

    Haynar Administrator

    Messages:
    3,637
    It sucks!!!!
     
  6. Ghakim

    Ghakim Well-Known Member

    Messages:
    169
    Yes thank you robregen and the rest of the devs for all that you do.
     
  7. hrududu

    hrududu New Member

    Messages:
    7
    Thanks to the devs for the great work you are doing!
     
  8. Lenas

    Lenas I Feel Loved

    Messages:
    2,968
    Nuke it from orbit and start over. It's the only way to be sure.
     
    Elrontaur likes this.
  9. Sturmm

    Sturmm Well-Known Member

    Messages:
    316
    Hudson: "That's it, man. Game over, man. Game over, what the f*** are we supposed to now, huh, what are we gonna do?"
     
  10. Elrontaur

    Elrontaur People Like Me

    Messages:
    1,291
    Let's all gather around the camp fire, sing a couple of songs...
     
  11. Elrontaur

    Elrontaur People Like Me

    Messages:
    1,291
    Also, now is the time for all hardened testers to show their mettle. Get in there and help us find the issue. The more people testing and submitting bugs, the quicker we'll resolve this issue.

    The only place my toons have lag problems is in PoK, so it's hard for me to test the LD thing. My connection is always solid except for Pok.
     
    Last edited: May 12, 2015
  12. Elrontaur

    Elrontaur People Like Me

    Messages:
    1,291
    Rob, how far back can you roll? We might need to go a few iterations back in the code to find the last working state of the server.
     
  13. Haynar

    Haynar Administrator

    Messages:
    3,637
    Anything weird u see going on. Conditions that cause lag. Or if u see lag or go LD when u do something. We need that kind of info.

    Thanks

    H
     
  14. Elroz

    Elroz I Feel Loved Staff Member

    Messages:
    2,283
    so i did some testing and im not 100% sure on this but it's a theory, it seems if i have a corpse in any zone, it causes a big lag spike when i zone that character, and the lag affects everyone in the zone. I zoned level 1 alts in and out of pok for 30 min and had no lag spikes, then i decided to go and get the lvl 1 killed, bound in pok. When he zoned in from death there was a bad spike, then i zoned him out and back, another bad spike, then i zoned both to qeynos and when the guy with a corpse in steamfont zoned to qeynos, another lag spike happened. This is why there's always bad lag spikes when someone is rezzed, they have a corpse.

    Also it might explain why pok is so bad, since so many are bound there and zoning in from deaths would lag the zone. Hopefully this helps
     
    Last edited: May 12, 2015
  15. Haynar

    Haynar Administrator

    Messages:
    3,637
    There is an issue with backing up corpses. Maybe it lags everyone out, then they get slammed with all the data trying to catch up.
     
  16. Elroz

    Elroz I Feel Loved Staff Member

    Messages:
    2,283
    Yeah after those bad lag spikes, it was desyncing me about 25% of the time. So can imagine if 3boxing and hit with those spikes it would be 3x worse and higher chance of desync.
     
  17. Mascha

    Mascha People Like Me

    Messages:
    553
    Nice work, Elroz, perhaps you are on track with the corpses. I am at work now, but would be willing to test when I come home tonight. What exactly should I test? Make level 1s and die? Which zones are affected mostly? POK and HHK? My problem with testing is that I LD on zoning about 75% of the time, so it is hard to peproduce the recent problem with the network and the original problem which I always had.

    Any tips would help.
     
  18. Cutty

    Cutty Well-Known Member

    Messages:
    128
    When I was on earlier people were testing that theory over and over and over again, lol... it was definitely causing desyncs and lag spikes but just as one person would be content it was causing it.. another guy would start, lol.
     
  19. Pinoit

    Pinoit Well-Known Member

    Messages:
    294
    I will gladly keep testing, and want to say thanks to everyone for keeping at it, developers and players alike! This emulator server is unique in terms of what it is--there's nothing else quite like it!

    I have a few questions for the Developers.
    • Can you think of anything (other than code in a patch) that changed between May 6-8 (last Wednesday to Friday) and today? For example:
      • was any hardware changed or is it showing signs of failing (assuming you have direct access to it and it's not remote)?
      • what OS updates, if any, were applied last week, particularly between those two days?
      • is there any line maintenance being done near or around the server location?
    There are a lot of people of technical profession (or enthusiasm) playing this game. It's possible it is not related to the game as you say. It's also possible that we may be able to help you puzzle through the problem. Don't be shy to share anything you can think of!
     
    Last edited: May 12, 2015
    Elrontaur likes this.
  20. Elrontaur

    Elrontaur People Like Me

    Messages:
    1,291
    Just out of curiousity, does anyone remember exactly when this started? Was it post the most recent patch?
     
  21. Haynar

    Haynar Administrator

    Messages:
    3,637
    There have been issues with some people going LD while zoning for some time. It has been baffling, as some are also unaffected and don't experience this so much.

    The guess was on the 5/6 patch it started. But rolling back to 4/29 did not fix the issue. So that makes me think its been around since before then. And may have just gotten worse as there are more corpses out there.

    H
     
  22. Elroz

    Elroz I Feel Loved Staff Member

    Messages:
    2,283
    I'm curious, how does TAKP magelo update? I know it updates basically after every mob I kill. I've seen the corpse tab on there although I don't have privileges to use it. I guess I'm wondering how tied into the server is it? Is it using some of my bandwidth to update that often? Is it what causes little lag spikes after I get "character saved." message? Just trying to come up with ideas
     
  23. Dranthor

    Dranthor Active Member

    Messages:
    162
    I have been fairly stable before and after May 6th when zoning, but my connection has been significantly less stable while in a zone since then, particularly PoK (including issues where I can't see characters and can't buff which started May 6). I haven't noticed any differences with or without a corpse for myself, and don't know how many are in the zone but I've generally been in low or unpopulated zones outside of PoK. Typically I have had no corpses during this time.

    I notice extreme data rates when logging characters in or when saving characters, often resulting in a disconnect of at least one nearby character whether that character was the one that saved or not. The issue is magnified while raiding. If I had to guess (based on no knowledge of the information sent while loading or saving), I would say there is a large amount of data that isn't all making in through before it times out, which is also being broadcast to all nearby (possibly zonewide) clients. If there was a way to reduce the amount of data sent and received during the loads and saves it may get the spikes into a tolerable range and improve the connection stability. [Would reducing the data rate help by smoothing out the data spike?]
     
  24. Haynar

    Haynar Administrator

    Messages:
    3,637
    I have noticed the lag spikes on Save()'s.

    I am not sure what else they allow access to the db.

    H
     
    Last edited: May 12, 2015
  25. Elrontaur

    Elrontaur People Like Me

    Messages:
    1,291
    I'm sorry but I just have to ask these dumb questions, where there any hardware updates to the server at all during all of this? Any software OS updates? It's always good to eliminate hardware and or OS software as a culprit.
     
    Pinoit likes this.
  26. Elrontaur

    Elrontaur People Like Me

    Messages:
    1,291
    Missed your post Pinoit, you are spot on.
     
    Pinoit likes this.
  27. Mokli

    Mokli I Feel Loved

    Messages:
    2,460
    That other server made this change on 4/25: * Downgraded MYSQL Version to fix some issues.

    Not sure if it could be related.
     
    Last edited: May 12, 2015
  28. Elrontaur

    Elrontaur People Like Me

    Messages:
    1,291
    Would it be possible to load a restore from prior to 4/29?
     
  29. Mascha

    Mascha People Like Me

    Messages:
    553
    I do not know if this is related, but since the update sometimes one client freezes during entering the world and then I cannot use the mouse on the other two clients any more. I then have to camp the clients that are online, but the window, where entering the world didn't work, cannot be closed by ALT + F4, but has to be closed manually via task manager. This is on a Win 8.1 PC.
     
  30. robregen

    robregen Administrator Staff Member

    Messages:
    8,386
    well, I've sent a PM to the person that provided the host for us regarding to Network / Hardware configuration. In the meantime, we could rollback a bit further like maybe on 4/22 or perhaps 4/15 to determine if that could possibly not be related to code or the game database.
     
    Elrontaur likes this.