Monthly Archives: April 2011

Apr 28

Jabbakam’s recent downtime due to Amazon server outage

It’s been nearly a week since Jabbakam suffered downtime due to Amazon’s outage. This episode taught us a lot of lessons and over this last week we’ve reevaluated our setup so Jabbakam hopefully won’t be affected by any similar outages in the future.

This is a post to explain to our customers what happened, what we did about it, and how we are going to be changing things in the near future.

The Problem

About 9:15am UK time on Thursday 21st April Skynet began it’s attack against humanity , targeting the Amazon AWS servers that Jabbakam runs on.  Our System Administrator Paul began to receive a flood of emails and texts telling him that our FTP servers and websites had lost the connection to our central database. Because our multiple servers could not log into the database, they began to clog up with running processes which, in turn, caused more problems for us as we started receiving emails and texts informing us that each server was reaching its capacity.

The end result of our database being made unavailable was that no cameras were able to save new uploads and customers were not even able to log into the website.

Our fix

Information coming through from Amazon was very sketchy, but we eventually figured out that the problem was not the database itself, or any of its connections, but that it was due to our central database which is stored on Amazon’s EBS. It turned out that the EBS was the primary failure point in the blackout.

As Jabbakam data is critical to the service we provide, we always run another database whose only job is to mirror exactly the main central database. This database is not held on the Amazon EBS. Our eventual fix involved converting this database to become the new main central database, and pointing all other servers to it.

Once we had done this the system was back up to normal and service was restored.

Consequences

It sounds relatively trivial now I’m writing it down, but at the time it was anything but. In total Jabbakam suffered about 7 hours of down time – our last clip being uploaded at 08:33GMT and the first clip after the fix being uploaded at 15:26GMT. This in itself was a minor success as the Amazon EBS (and many other sites using this utility) was down for
almost 30 hours.

However, 7 hours is still a long time to be down and we have to make sure this does not happen again. As such we are changing the way our server architecture is structured. We are going to have less points of failure but also plans in place to circumvent any problems if we detect them. We’ll also be more geographically diverse – if there is a problem with our servers in one location, then we’ll fire up Jabbakam somewhere else.

What does this mean for our customers?

Whilst we appreciate our customers suffered a serious chunk of downtime with Jabbakam, you should not notice any further disruption of service whilst we make our changes. Our system is pretty durable most of the time so we simply need to make sure we increase this durability. There may be a tiny bit of scheduled downtime but we will keep this to an absolute minimum and we will warn users in advance on the Jabbakam twitter feed.

As always, we’re very keen to hear customer feedback, so please do let us know if you have any information to share regarding this recent episode.

Apr 27

If only Michael Buble had had Jabbakam

Camera surveillance made big news earlier this month in the most unexpected of places, at the altar!

Canadian Crooner Michael Buble who hit the big time in the UK and around the world with his smooth old time singing style found himself in the middle of a very modern predicament when he got hitched recently in Buenos Aires.  While he and his Argentine bride were tying the knot and celebrating with family and friends, burglars took advantage of the situation and broke into their home after cutting alarm cables and covering up video surveillance cameras.

While we here at Jabbakam certainly sympathize with the happy young couple, we know that had Michael had the Jabbakam video surveillance system with motion detection and SMS alerts set up to alert his mobile phone or to alert his camera contacts to new camera activity, the crooks might have got caught before they ever got started.

Event triggers, like motion detection available in most IP Cameras are a cost-effective crime fighting tool as are wireless capabilities. Without wires there are no cords to cut, and with motion detection and SMS and email alerts, any motion (like someone covering up lenses on security cameras) will automatically set off alerts which are sent to you… and/or to your trusted Jabbakam camera contacts, who can be watching your back when you’re busy doing other things.

Apr 13

Around the World with Jabbakam and IP Technology

Just last week an article in the Toowoomba Chronicle in Australia reported on the small Queensland  town of Tara where recently approved legislation by the regional council to install IP security cameras around the city’s “hot spots” is big news.

IP camera-based security systems are working their way into every corner of the globe, creating new markets or replacing analog CCTV systems — and we here at Jabbakam are banking on the fact that this trend will continue as businesses, private citizens and local city councils start to look for flexible online video surveillance solutions in the IP camera market.

Here is a sampling of a few other places around the world where IP Camera surveillance systems are finding a home. Have a look!

On the studio lot in Hollywood

In Kenya for emergency services

At the Delhi International airport in India