Friday with Means - June 26th, 2009 - The dead awaken and consume coffee.

[Means]

This update was the perfect time to release a /facepalm animation. Most of the team is slowly recovering from being awake for almost 48 hours straight.

There were a great many things in the update that we thought might destroy the universe.

A new client RDB for everyone that allows us to add more files and process faster.
Instanced cities for everyone.
Instancing Pandemonium.
Instanced playfields finally being displayed with the correct lighting and effects they were always meant to have.
Messing with the City Advantages code.
New apartments accessed in a new way.
New Hostile Target nano window....

Any of these could have done it...as easily in the past it could have been the social window...the mechs...the battlestations...or the raid system...or the new instancing system itself. So many opportunities to break the entire game...and force the rollback we have managed to avoid for about 6 years..since 12.6.

What actually did it?

The new system for cool-downs for nanos. In the past if we wanted an action to be unavailable for use (beyond the recharge/global cooldown of everything) we were forced to depend on silly blocker nanos..."You can not cast this while XXXXX is running." The new cooldowns meant we could dispense with the silly blocker nanos and properly display (on the shortcut bar even!) the amount of time until these actions were available again. This is where things went "bad".

"Crashing" or exiting the game very rapidly would break these cooldowns and corrupt your character information. Anything contained in your character after this cooldown information was corrupted...this resulted in your research and perks effectively vanishing. Importantly your "Save" position also became corrupt...anyone who died in this state would then crash the entire server...not surprisingly breaking other characters on the same server who were also in this cooldown state. One broken character would then create more crashes...then more broken characters as a result...and from there it began to spread across all dimensions like a virus. Doctors had the longest cooldown of this kind in common use...so they were immediately the first players who were susceptible to this issue...and in the end what allowed us to locate the problem and eliminate it in the update we released at 5am on Thursday. The sheer number of characters affected by this issue left us no choice but to roll back the entire day...with no readily available way of detecting this kind of corruption in the database it was far preferable to months of crashes as these characters appeared on the servers and hurt them again....and other characters by crash issues that will always persist (ie: lost items due to server crashes).

I am very sorry it took as long as it did to locate the issue...all the posts from the community on this issue were instrumental in helping us locate it as quickly as we did...the sheer horrible random nature of this issue (in terms of the symptoms of vanishing perks and research) made it very hard to identify. We are very lucky to have the talented and dedicated coders, designers and operational support teams that we have otherwise this issue could have been plaguing us for a much longer time.

The population on Test was also incredibly helpfull in helping us put an end to this. It is also not fair to say that anything done on test or not done on test could have prevented this issue. Only a server crash during a precise 10 second window would have produced this result...and that is not the fault of the test population...they were extremely helpfull in making sure the other significantly risky improvements to AO in this update ran as smoothly as they did and we are very gratefull for their help. Thank-You Nirvelle for lending us your account! Nothing short of a huge live environment was going to expose this weakness...While I am sorry it happened at all I do think that it could not have been repaired as quickly or as well as it was in any other conditions.

Stepping into the AO universe and making the kinds of changes we have been making for the past year are always horribly risky. I also like to think we are really beginning to make progress...and despite the loss of 11 hours this week I feel like we are starting to make good progress. Again I am sorry that any of the pain and suffering we normally deal with when combating the old issues of AO spilled out onto our players. I hope we can enjoy another 6 years before this kind of issue gets us again.

I am going to choose to see the positive aspects of what we are all achieving together in making AO better rather than dwelling on this incident and letting it scare us away from making the dangerous changes that are necessary to make AO the game it always should have been. I love AO now...but it can be so much better as long as we never give up. We'll get there.

This week despite disasters:

Macrosun still found time this week to start work on an in-game mail system.
Genele continued work on the 150-200 new instance.
More balancing work and meetings abounded...we are just getting started.

Thank you to all of you who provide the support and inspiration to the whole team to make the super-human effort required to run AO worth every grey hair. Making us laugh at 4am wasn't easy...but you managed it. Thanks again.

Have a great Anarchy Online Birthday! Thank you all for helping us make it to an amazing 8 years. I'm looking forward to the next 8!

/salute

[Ittledoc]

1st

w00t first time ever!

"Macrosun still found time this week to start work on an in-game mail system."
Nice!

PS: I really think AO has loaded faster and zones faster now

[Ekarona]

nth

Atleast I got references of new apartment stuff, without wasting 989M not that I would have any use of that cash ;D

You made lot more sense than I did recently ;D

[vadagget]

3rd!

[Carlmalone]

I love the new database.

[Cdogg421]

oh u want a laugh ey. how about this

[Chrys]

/drool

Is there any chance the new global cooldowns can be modified to work with our tools? There's a rich vein of possibility for fixing MPs casting issues with this...

[Graftmage]

Originally Posted by Chrys Is there any chance the new global cooldowns can be modified to work with our tools? There's a rich vein of possibility for fixing MPs casting issues with this...

SS is meant to block all casting

[RedR053]

rawr

[Darkripper]

gief more instances plx!
Make that new instance interesting, with some tactics, not tank n' spank... a few different mobs would be fun also, sick of killing always the same faces :P

[krumpfwild]

Originally Posted by Means Thank-You Nirvelle for lending us your account!

OMG ! Account sharing !!! Ban Means ! Ban devs ! Ban them all !!!!


Oh well, great work on this patch, not everything is perfect (i'd even say the opposite) but it's good to see the A-team.. er... AO team still invest a lot of efforts to make our game better.

This message is approved by Krumpf.

[Rubika-1]

Happy B-Day AO Big Thank You to Means and Crew getting it all worked out for us, and putting in all the overtime... My God, did you guys really stay awake 48 hours to work all these problems out? 0.o All of you deserve a big fat bonus for all you do. Looking forward to seeing all the new comming changes, new PF, mail system, ect... These really will help out alot.

Take it easy over the weekend, you all deserve a much needed break.

Cdogg, nice vid there made me laugh

[Dancingrage]

Since you replied to the one thread regarding revamping the noob experience I'd like to see more on your take on that. It's a good thing you put up that you'd seen it, a lot of us are worried about player retention and how that's starting to affect the game population.

[Chrys]

Originally Posted by Graftmage SS is meant to block all casting

I know you NTs love that in order for your nemesis to survive, they have to disable all possibility of making your life miserable <3 But we'd like a second opinion plx xD

[ArienSky]

Thank you for completing the angel outfit (Veteran halo) and the new white outfit.(and light robe, working in social)

The apartments are a nice addition, but where is teh orbital bar?
Another birthday surprise on the way?

More documentation on the GUI issue though, please

Kind Regards
Ariensky

[PhoenixOfAges]

Originally Posted by Means Any of these could have done it...as easily in the past it could have been the social window...the mechs...the battlestations...or the raid system...or the new instancing system itself.

It was the BSs, they were bs to begin with and look at what they just did now, toss em.

Originally Posted by Means What actually did it? The new system for cool-downs for nanos.

No, it was the BSs darn it.

Originally Posted by Means Macrosun still found time this week to start work on an in-game mail system.

But you said you didn't like those. Be more useful if they applied it to player shops so GMSs allow us to search and receive things. Would even make shops in instanced cities possible I image.

All the same that sounds handy and is appreciated.

[SoldiTwo]

Originally Posted by Means ... The new cooldowns meant we could dispense with the silly blocker nanos and properly display (on the shortcut bar even!) the amount of time until these actions were available again. This is where things went "bad". ...

A note on the cooldown.
I wrote already in another thread but, I used to have my AO configureed so that the recharge time was only displayed on the actual nano I casted and not on every nano icon in my shortcut bar, the others just greyed out.

Now I have the number on every single nano icon.

The only possible configurations now are:
1) all grey out with or without fade + timer on all
2) all grey out with or without fade and no timer at all
3) etc. ^^

BTW
Good work all(Testserver included)!

PS
On the road to a more WoW like AO, please don't remove the character skillings

[Chrisax]

I remember 12.6 (Oh my God... am I that old?!) and honestly you managed well today.

Your current team is progressively fixing things that we didn't hope to be fixed or improved anymore (like timers on nanos this time). We have been often told in the past that touching the legacy code for anything else than minor tweaks was not something that could be seriously considered... but you are doing it. You already did it. You want to continue... and even rebalance professions (something else we were told long ago that could never happen in-depth). OK, the least we can do to thank you is to grant you the right to cause Apocalypse once a year! (I said once!)

It's good to see some old aspects of the game finally polished and working as they should have always done, like, once again this time, the cool-down timers on nanos.

I hope that, the same way, when the new engine is about to be released and massively advertised, you'll be able to improve the game for newcomers, make it more entertaining and also make the learning curve more newcomers-friendly. (And get rid of so many misleading and confusing features, weapons, items that make no sense now, and made little anyhow in the past.) But that's another topic.

On that day, though, make a social T-shirt and sign for us "Newbies welcome here" that we can wear or hold to tell new players we are ready to help them and they can talk to us.

By the way, off topic again, do you think you'll have something to show or tell us soon about the new engine? I need badly to update the info page we made about the old new engine...

[Rubika-1]

Originally Posted by Chrisax On that day, though, make a social T-shirt and sign for us "Newbies welcome here" that we can wear or hold to tell new players we are ready to help them and they can talk to us.

This please Nice idea Chrisax!

[Mekh]

Originally Posted by Means ...snip... There were a great many things in the update that we thought might destroy the universe. ...snip...

Hmmmm....



Thanks for the update... Lotsa exciting stuff.