Dear TAK Developers, GMs and Staff: Hoyas. I'm Zetro, a former player from Alkabor and officer in the guild Temerity. Congrats on the progress you're making with The Al`Kabor Project. I miss Alkabor. I especially miss what the server was prior to its final year. I don't think I'm alone in that sentiment. For myself and for others who are interested, would you mind having a discussion about security? Your stickied post of Server Rules is SUPER encouraging. I love everything I read there. No client hacks. No manipulation of game data in active memory. No scripts. No RMTing. The civility stuff. It's a fine constitution for a new server. What I hope you're willing to talk about is your commitment to those standards. Is staff going to enforce those rules? For all I know it could just be a boilerplate document that you use on new servers. And there'd be nothing wrong with that. Of course you get to create any kind of server you want. I'm just trying to figure out if it's a match for people like me. Besides your willingness to enforce those rules, I'm also curious about your means to enforce them. You're allowing the PC client. I don't hate PCs or anything culty like that. I was a PC user that got a MAC for Alkabor. But the PC client was the thing, combined with accounts becoming free, that killed us. You have to know that to know the server you're emulating. With the PC client came macroquest hacks and 3rd party EQTrainers that could hack active memory. With unlimited free accounts came no fear of banning or accountability for using those things. Ok TAK, you're specifically allowing MQ developer EQmule's version of the pc client. Meanwhile, The Alkabor Project is an EMU. So accounts would be free. With those two ingredients in place, can you talk a bit about why and how this will be different? I've nothing against playing on and alongside PC clients if it doesn't enable the cheats it did on AK. Thank you for your time. Zetro
Secrets is a dev on this server and is the person that originally brought us the PC client, so if that worries you, that may be a problem. However you should also realize that AK had zero GM staff. The dev staff here is very active and I am 100% confident they can detect when someone uses programs that they probably had a hand in writing. The PC client brought a lot of good players and people with good attitudes. A few people were vocal assholes and started the PC vs Mac rhetoric, but there's no reason to exclude the entire player base. Also, the fact is, it was possible to hack EQ from the OS X side anyway.
I appreciate this well thought out post. I would like to start out with saying that the emu servers don’t have a form letter base for a rules template. They are very similar to PeQ’s rules as I copied the rules post and heavily modified that to suit our needs and the needs of the players on our server. We will actively adjust them to suit situations that arise. As far as enforcing them goes, we have a mixture of people on staff here that have a background in management and military. Both of which will have a direct impact in how we enforce and manage this server. We are very willing and able to take each situation as it comes up very seriously and react according to the given circumstances. Being that this is a player based server (emulator), we have direct control of what occurs server side. We can, and have tailored tools to more effectively manage and enforce the needs of the server and community. We also do not report to a company, shareholders or a “boss” as far as what we do with this server. We do this as a hobby and for the most part I think some of us prefer to code and manage the server over playing on it. But don’t mistake that for us being disconnected from the player base, as we will likely play as well. We are not 9-5 limited. I can’t fully speak for the others on the team, but I know I willingly put more time per day on this project than any job I have ever worked. As such I take a special seriousness and pride in making sure this work is not wasted or ruined by those that have no problems destroying others. The amount of work put into this server specifically by our team here is astounding. Now we have avid player testers putting their time into making sure the server runs right and is accurate from the player perspective. We can and will do everything within our means to make sure that it is not all wasted or ruined by a select few. No single player, connection, account, character, or hack is greater than the whole of the project or player base. This will be reflected in the seriousness in our approach to issues that harm this community. Don’t mistake any jolly nature in our staff as a willingness to let things slide. I share your sentiments about the current PC client. We are aware and considering a PC client change that would more effectively address balance between mac and PC gameplay issues as well as improve the ability to block known hacks. For the security reasons you put forth, I can’t really discuss our approach on hacks and cheats. We also are aware of OSX based hacks too. Rest assured, we are aware of many and will do what we can for damage control in this area, as well as a very active and willing staff that is nearly around the clock available for in game issues. We are considering a PC side client change. But that is in it’s beginning stages. We have to work from the ground up on compatibility. When the time comes we will announce a “sunset” of the eqmule’s client support to allow for a testing transition phase of the new implementation. We can and will specifically block certain clients. For example, there is no way modern clients can connect unlike many other emu servers. Signed and proof read by, The Al’Kabor Project management team.
Well a major reason we are supporting the pc client is for the numbers. So after brain storming, we decided to follow in the true Mac user spirit. We do not want to discriminate against players due to platform or equipment. I personally think every one should be able to enjoy the experience that we all had that was taken away from us in a less than desirable manor. Limiting the client would keep numbers a little lower. We have no gain from having 5 players to 500 players other than the fact that we preserved a small piece of history that brings some enjoyment to some one, basically a job well done. As far as security goes: We have tools in place to curb hacking, and we are constantly testing cheats and closing up the loopholes. Im sure sony had the same thing but I think they fired the fat guy eating doughnuts that looked at the hack logs and the "camera footage" , during some crappy downsizing. Since we don't get paid and have no shareholders that can never be a issue. A lot of our staff are and will be players so we actually see whats going on and I know I have stated it before but the rules are there to keep the server from looking like Al Kabor did at its close. You never know, you might be playing along side a administrator and not even know it. Also we have GMs that are very active, some are extremely disciplined and will just about issue a suspension for dropping items on the ground as they see that as littering. GM's have been present during the alpha testing thus far. Players, I'm sure have seen that GMs have tossed out some heals and buffs and have been interacting with the players when there is a problem. Plain and simple if a GM see's you doing something wrong then you will probably get banned. I know that with the characters I had on AK, I was very unwilling to risk losing them so I hardly think many folks are going to want to " start fresh " and then get banned 7 ways to sunday. This also means that if players have evidence of some one using a cheats and or hacks of any form they are welcome to present it in to the staff, though I urge people to use PM's for this due to the sensitive nature.
All the work the devs do is out in the open to each other. Anything that is to be applied to the "live" server is only from the publicly viewable git and svns. Our staff does take steps to make sure that any submissions have the entire project in mind. As far as concerns for Secrets specifically, the only people that access the server and have direct influence on it's contents are myself, robregen, and Jikasoz. All changes and updates will only be applied to the running server by robregen and myself. Cavedude and Secrets work almost entirely on server code that gets submitted to git only. To be honest if it wasn't for BOTH of these two, this server would never have existed. To give a little story behind the structure of this project. This server is one of potentially many merely using the PEQMac server code that Cavedude and Secrets submit to. We are considered the "Official" server yes. But we are a separate entity.
I also strongly encourage PMs to the GMs if questionable acts are seen in game. PMs on the board are likely better than in game as those reports are saved and a matter of record. If everyone takes part in making sure these hacks and bad behavior are noticed and reported, it will help us to make it safe for legit honest play server wide. No matter what tools are made for an emu or a commercial game, there will always be something that "gets through", at least temporarily.
Being a private server without all the hierarchy of multiple layers of management really helps. I have played on PEQ:TGC the last year and the GMs have been great. Some drunken jerk being racist in /ooc didn't even last 24hrs before he was begging on the boards to have his account un-banned.
Re: TAKP Security Speedz and the Project Team: Thank you for the thoughtful reply. It's encouraging to hear that the staff believes in those security ideals and intends to uphold them. The willingness part of my question is answered. Because I take you at your word. I wish you could speak a little more about the means. I understand we can't talk about specific anti-cheat measure without making those measures easier to defeat. I don't want that. Let me ask a couple questions that are more general. Do you anticipate being able to detect and/or counter the passive types of cheats as well as the more active stuff? I hope so. I hear that the P99 folks have developed good detections these days. Will your team get the hook-up on some of those processes? Zetro
Re: TAKP Security Short answer to that is no. Partially because they are closed source (although if I asked Rogean and promised to keep it closed I am sure he would help us.) But the main problem is the client. P99 uses Titanium, a newer client. The newer clients are more dependant on the server telling them what to do to prevent cheating. That helps them a lot. Unfortunately, our client does a lot of things on its own. Often times, it doesn't even reach out to the server at all. More often than that it has no internal security or checks to see if somebody has forced it to send a packet. If the client is hacked and sends a packet that the server can't distinguish from a normal one, it may allow it through. But, these situations generally are minor hacks. All the major things the server will put a stop to. For example, you will not see level 1s in potime on this server. The server code simply will not allow it. Also things like items are heavily watched. The alpha testers are dealing with item desyncs which I am currently working through. We don't actually have to have item desyncs. That is an EQEmu invention to make absolutely certain that the server and client agree on what item exists where and how many there are. Our default behavior is to delete if the server doesn't like what the client is telling it. There are bugs with it now, and it's triggering false positives especially on Intel, but I'll get it working smoothly. We also have anti-MQ2 code that can be tweaked to work with any hack. If somebody is cheating, they'll generally be silently logged. I also have plans for closed sourced security measures too. I am generally an open source hippy, but security software is the one exception to that. The foolproof way of course is to write a process watcher, to tell us everything that is running on a system. But, I won't write that or be apart of that. Privacy is far more important to me than if somebody is using a speed hack. I also won't write anything to log private /tells. But fair warning, public channels including guild chat will be monitored.
Cavedude, Brell bless ya. I appreciate the details but worry it may be too specific for a public forum.
Re: TAKP Security On the more general topic I'm not as anti process-watcher as you as long as it unloads when the mother app does, logs processes and nothing else, and users are fully advised of its presence. I don't think I mind system logs for tells, either, since they travel through the server. Not to mention that we caught a scripter on AK who passed parameters to his bot through tells. I respect your values on the issue, though, and when in doubt yours is the right side to err towards. Zetro
Its also important to remember that most of the p99 stuff is now spyware they run on your machine. And that really irks me.
Yea I am with CD on this one, not too keen on loading a process snooper. Has too much of a Blizzard feel to it. As far as tell monitoring, I am against logging tells but not against monitoring. If someone gets reported for harassment and w/e other things in private tells I think GMs with a higher status rank should only be able to see tells. A couple of extreme examples as to why: There have been plenty of stories covering companies/services online getting nailed for not enough protection to monitor behavior against women/minorities/children. As it stands, we need to have a way to control/monitor/prove this and take action accordingly. 1) a child gets lewd and explicit tells every time they log in. 2) someone gets constant threats and aggression in tells ruining their time on the server. 3) a woman get harassed in the same way as in #1 Then the victim logs off to never play again and the attacker moves on to another. I am sure there are many other examples that can be used, but these are the big 3. Also forgot to mention, since SOE GMs have been known to have the ability to do this, and the simple fact that this is a community made and ran server(open source). There should be no expectation of 100% privacy in all interactions with the server. Would we be public about the dirt we would potentially see? Not a chance. Tho, we can not guarantee that the "middleman" in the internet chain of things is not able to snoop our packets. As certain gov't agencies have shown time and again lately.
I'm pretty big on privacy and a huge Snowden fanboy and whatnot, but there are ways to implement some process scanning while maintaining privacy. Instead of sending back a list of processes (which I also would not want to do) you could send back hashes, or better yet merely a boolean yes/no if a bad process was found from a list of known bad processes. Even though I don't log into p99 outside a VM or a machine with nothing on it, I would like to see some sort of client side cheat detection implemented here, even if it's open sourced and the limits are explained to users, because I assure you the cheaters will come out in force. Project 1999 actually implemented custom encryption and the cheaters managed to break it and make showeq work. The fear of it existing will discourage most of the script kiddie cheating, and when the more capable cheaters see a new client patch go up, they will again have to shut down their operations for awhile to check it out. Even the smart cheaters will make mistakes every now and then, and the more hoops you make them jump through, the more likely they will be to make a mistake.
You also have to remember due to the restrictive nature of osx and sandbox execution they make it harder to get a full processlist. TLDR; if you turn off app sandboxing you forfet a lot of developer tools and frameworks.
What I know about internet security wouldn't fill a tinkered thimble. If you can alleviate Zetro's concerns, then I will be satisfied !
For my two cents: I don't personally think it's a net win (in terms of effort:reward) to go after the local stuff directly with process watchers and all of that. A simplistic check of the reported client is nice because it stops non-malicious people from somehow accidentally using the wrong client download or something. But beyond that, trying to suppress the possibility of local hacks is pretty futile. Even if one ignored the privacy issues, it's an arms race type of situation that burns a lot of time, and it's one you ultimately can't win outside of some sort of hardware trusted computing platform that doesn't exist and that you wouldn't have keys for anyways. I've always thought that the correct answer to those concerns is better server-side monitoring. A lot of client-side hacks result in related server-side action patterns and database update patterns that are easy to detect back at the server if they do anything really significant. For the few corner cases that might be hard to catch routinely with triggers and alarms, active, involved GMs that know the community and snoop around the zones from time to time should be able to ferret out what's happening (and again, without ever really touching local security on the client itself).
Very good reply Cabot. This is pretty much how I approach it. The server side checks are the best way to go about it IMO too. Along with the active GM presence to monitor and be involved with the community. That is something we didn't have for a long time on AK. This isn't going to be a server that is put up and ignored to be allowed to grow wildly like a weed.
I disagree with the assertion that client side checks are futile. One can make the argument that it's not worth the time investment, but client side process scans can be effective. After project 1999 implemented it, they caught hundreds of cheaters. I know the list of caught players that was leaked was real because they deleveled several of my friend's accounts. Blizzard has warden and dozens of first person shooters use punkbuster; if it didn't catch anyone they wouldn't use it. Right now you can download already working copies of myseq, mq2, and that 'trainer' program for this server with installation and use instructions a goggle search away. Can anybody provide me a link to a working project 1999 client hack? Or even one released after their cheat detection was implemented in 2011? You really can't deny that cheating on p99 declined dramatically after their client side checks were put in. Yes the most technologically capable and determined cheaters will bypass the client checking eventually, but they are motivated to not release their creations to the world as it would potentially make them detectable. Most cheaters secretly wish they couldn't cheat-- the anti-cheating efforts from the p99 team have high approval from players. Secrets wrote some of project 1999's cheat detection code, and if P99 were willing to share some of what they have, development time could be negligible. Of course there are ideological reasons to not want closed source code or process scanning of any sort to be involved with this project which would prevent or limit use, and I respect that, but lets not pretend it doesn't work. Server side checks are of course the best countermeasures, but they won't catch everything and some checks can be cost prohibitive CPU usage wise. Doing both would catch the most cheaters.
Is that still undetected by the process scanning p99 uses? And if so, is it because p99 staff refuses to pay redguides for private access? Even if it isn't, 2 years is still a long time without a public hack. If p99's anti-cheat started catching that soon after it was released, it only proves my point; it can't stop them until they are made and released.
Appreciate your words on this, Torven. And I agree. It isn't futile, and it is worth it. That redguides link is over a year old without replies or updates. P99 may have handled it in stride.
He asked for a link about post 2011 (2013 fits the bill) hacks. And fwiw that hack has been front and center in RG for a while, and of course that post hasn't been updated they would update the download post in level 2. Hackers will always find a way if they have the will; however, if the community is vigilant in addition to active GMs we can keep most of it out of the server.
Also remember what Stairs said about the OSX sandboxing. It's a lot more difficult on OSX than it was on PC to do the client side monitoring. Also, focusing on server side monitoring protects all clients, regardless of OS, without the security/privacy implications of client side monitoring. All that said, I think when it comes to overall concerns about hacking/cheating/exploiting, is to not underestimate the impact of having an actively developed server. I think the discourse is good, and I certainly don't blame everyone for being concerned, but the devs and the support team wouldn't but the time and effort into this server to let it have no substance due to hacking. From an overall perspective, the team will be monitoring all aspects of the environment to counteract nefarious actions, monitoring things like the economy - bazaar prices, RMTs, and other trades to make sure it's kept healthy. We also have a number of checks on the DB to watch for item dupes and other exploits, let alone the code built into the server to protect against it. Like others have said, ultimately you can do whatever you want on the client, it's up to the server to accept it and store it, so from my personal perspective, that's where I would like the devs to focus. It's also the most efficient from a development and administration perspective, as we don't have nearly as much insight into the OSX clients. We'll draft up an official policy on security and post it to help clarify the current position of the team, as always it will be subject to change, and we'll also do our best to balance transparency and discretion when it comes to the security methods used here.
I can also scare you all but in the name of full disclosure: The intel mac client is compiled with debug symbols still installed. What does that mean? Well Zamiel had a tool called ZamielQuest which would allow you to attach the debugging tool gdb to the intel client and call functions inside the client. There isn't much we can do to stop this sadly since this method of hacking looks exactly like the intel client (technically speaking it is). What we can do is detect warping and duping stuff server side. But there may be things we can't stop because of this. The good and bad news is that Zamiel posted his source code on github so i've gone over it pretty thoroughly and can see what its able todo. That being said i know faite had some tools too but the source for those has never been released.
Forcing hack users to buy a Mac in order to use them would still significantly reduce the amount of hacking going on though. It would also be somewhat ironic... I would strongly advocate putting up hurdles (client side scans) for hackers to have to jump over even if they don't 100% stop them, as long as it wasn't a massive time investment that would prevent work on important things.
