Monday, May 26, 2014
Saturday, May 24, 2014
It's been a bit of a worry for Linden Lab this week, suffering some of the worst problems on a grid-wide scale for several years.
This week users of Second Life suffered outages, logon suspensions, unrezzed avatars, extreme lag (even for them) and in short, some of the worst on-going performance issues in quite a long time.
In recent years it has become quite a rarity for Linden Lab to hold themselves accountable for any problems caused by bad code or hardware failure. Even when they accidently removed all megaprims throughout the grid a few years back, leaving gaping holes in every region in the world, it was passed off as something the engineers would work on and fix, without much in the form of apology, much less an explanation.
So it is somewhat a surprise that Landen Linden has written on the Second Life Blog and offered a full account of the chain of events that took the grid to grid-lock.
Hopefully this is the dawn of a new era on open and frank accountability from the Lab. They will probably find that if they explain what has actually happened, people will be more understanding of this kind of service disaster.
From Landon Linden, in a posting entitle, The Recent Unpleasantness, here is what happened...
In my first few months at the Lab, we removed more than a hundred major single points of failure in our service, but several major ones still loomed large, the granddaddy of them all being the core MySQL database server. By late Winter 2009 we were suffering from a core database outage a few times each week.
With a lot of hard work and countless long nights we stabilized the service and started making major improvements to the overall stability and performance of Second Life. However, despite our continued improvements, and the relative tranquillity they have created, the spectres of technical debt and single points of failure still loom over our operations. In recent weeks some of them have struck and disrupted Second Life. So much so that I want to explain the outages that have occurred, how we addressed them, and what we are doing going forward.
First, that core MySQL database cluster still exists. It is still the core of many of our central functions. When the write server fails it takes a minimum of thirty minutes to promote a new server into position. The promotion itself is actually relatively quick, but its numerous dependent services must all be taken down and brought back up carefully to ensure that they are all functioning properly.
In the last two months the core MySQL write database has hit two different fatal hardware faults, driving us to temporarily halt most Second Life operations. In some sense, two major write database failures close together is bad luck, but we cannot depend on luck to ensure the reliability of Second Life. In the very near future, we are moving the core MySQL write server to a new hardware class, on which production read servers are already running. Moving the write server will further improve overall database performance and make failures less frequent. It does not of course solve the root single point of failure problem so in the coming days, weeks, and months we will be reducing the impact of database failures even more. This includes continued improvement to the rotation process, extracting more functions out of the core database cluster, and further reducing the number of features that depend on the single write server.
The core MySQL database, however, has not been our only problem recently. A few weeks ago there was a massive distributed denial of service attack on one of our upstream service providers that affected most of their customers, including us, and inhibited the ability of some to use our services. We have since mitigated future potential impact from such an attack by adding an additional provider. There have also been hardware failures in the Marketplace search infrastructure that have impacted that site, a problem that we are continuing to work through. Most seriously though was this week’s four and a half hour long login outage.
On Tuesday morning, users stopped being able to get into Second Life. The root cause was created over ten years ago in a system designed to assign a unique identifier to the hand-off of sessions from login to users’ initial regions. At 7:40AM Pacific Time, that system quietly ran out of possible numbers to assign. It took us four hours to isolate the problem, test a fix, and deploy the change. Users could immediately log in at that point, but it took an additional two hours for systems to settle out. When tens of thousands of users rush back into Second Life following an outage, we have to deliberately throttle some services to prevent further breakage.
Having such a hidden fault in a core service is unacceptable, so we are doing a thorough review of the login process to determine if there are any more problems like this lurking. Our intent at this point also is to remove the identifier assignment service altogether. It not only was the ultimate source of this outage, but is also one more single point of failure that should have been dispatched long ago.
We want to apologize for all of the recent problems and the frustration they have caused. We too are frustrated and are intent on making our service better. Few things give me more pleasure than every day helping to make Second Life a happy and fun place. Thank you for your patience and support. We simply could not have a more devoted user base and for that we owe you better.
Thursday, May 15, 2014
"All you need is love" according to the Beatles... however at The Crown we also believe you need good company and great music... and who better to listen to than The Fab Four themselves....
DJ Jonesy shall be bringing you all their hits from over the years, as well as taking all your requests, so head on over to The Crown on Thursday 15th May from noon - 2pm SLT
It might be that you want to "Come Together" and "Twist and Shout" after a "Hard Days Night..".. so whatever you like.. "We can Work it Out".. so take "The Long and Winding Road" to The Crown and "All Together Now" we can "Help" one another have a fantastic night!... *groans!!*
See you there... !!
Wednesday, May 14, 2014
Are you a bargain hunter? Many of us in Second Life love a bargain, from the new resident, through to the oldy who just enjoys their freebies... and here in London City we have it all!
In London City Victoria sim lies the Freebies Mega Store which is packed out with the best freebies for all.. from ladies club wear to lingerie.. and gents suits, jeans and Tshirts it really has something for everyone.
The bargains however do not stop there! London has an amazing array of stores, and many of those also have some free items, so it is well worth having a wander and seeing what you can find.
At the time of writing on wandering the Victoria sim we also found the following:-
- Moooon offering free Gestures including one saying "I Love London"
- Western Rodeo offering outfits and a free rodeo pig and turkey!!
- RGDW for Men - free gift
- Creations by Alva has a vendor out packed with Dollarbies
Monday, May 12, 2014
Congratulations to Austria this year for winning the Eurovision Song Contest with their artiste Conchita, the bearded lady.
British entry Molly was Smitten-Downes with a sore throat just one day ahead of the contest and there was a worry that at one point she would use the rhythm method and pull out just ahead of the contest, but on the night, like the true star she isn't, she sang her heart out and sailed into obscurity securing just 40 points landing us in 17th position.
We last won Eurovision 17 years ago in 1997 with Love Shine a Light by Katrina and the Waves.
This is the longest gap ever for us since last winning. We've even tried rolling out the big guns in recent years with class acts such as Queen Victoria's favourite, the 79 year old Engelbert Humpadink and ever tuneful Bonnie Tyler. Some of us fear that until we wheel out Cilla Black on a gurney, we shall not win again.
Enjoy the video.
Sunday, May 11, 2014
Friday, May 09, 2014
You can enjoy the grizzly spectacle of the campest show on earth, The Eurovision Song Contest, in London City on Saturday.
From 12pm SL (8pm UK) get your space on the dance floor as we celebrate by-gone British entries, then listen to this year's hopefuls as they each croon their way into the inevitable oblivion that awaits them.
Our half time show this year features, The Brotherhood of Man and Bonnie Tyler!
Enjoy the show here in London City http://maps.secondlife.com/secondlife/London%20City/120/115/24
Monday, May 05, 2014
Sunday, May 04, 2014
About to start.. DJ Croft at Substation on the Road. Join us for a pool party and enjoy pop tunes from Croft's Time Tunnel.
From 9pm UK / 1pm SL its over to Bar Soho for great music and a quiz with Keif, Bexy and Katty
The Mad Hatters Tea Party at Juiced!
Sunday 4th May 2:30pm - 4:30pm SL
Mad hatter DJ Jon is at the turntable and the very sexy MereJaan is hostess for the event.
10.30pm UK / 2.30pm SL
Get your swimming costumes on and lets have a bank holiday paddle together with lashings of grog on this warm night!
Friday, May 02, 2014
So now you have recovered from the last bank holiday, it's time to do it all over again with the May Day Bank Holiday...
All the venues in London City are holding special events throughout the weekend all the way to Monday when there will be a special Concert in the Park.
Formed in 1976, Madness is one of Britain's most enduring bands still performing almost 40 years later.
Having spent over 200 weeks in the charts, they are perhaps best known for songs House of Fun, One Step Beyond, My Girl, Baggy Trousers, It Must Be Love, Embarrassment and Our House.
You can enjoy this brand new tribute show with us in London City this coming Saturday. The Party starts at 12pm SL (8pm UK) with the concert an hour later at 1pm SL.
Concert Area: http://maps.secondlife.com/secondlife/London%20City/123/110/23
This event is sponsored by DX Exchange, Linden Dollars at best rates
The Crown is hosting a Beach Party tonight to celebrate the welcommed return of our beloved DJ Crofty !!
So if you are fed up of snow, rain and fog, slip into some beachwear and get on over to The Crown where we will be partying in the sand as Crofty plays some great feelgood tunes.
Elle will be there to host and rub in suntan lotion (she is sooo helpful!) and there will be lindens to be won for best in beachwear. Head on over the party starts Friday at noon SLT til 2pm.