October 6 server outage - Post Motem

below is the post mortem report of the October 6 outage.



Our server had an outage at 13:30 GMT due to a high trafficked image. (http://beeimg.com/view/h8150456397/) according to the referral logs the image was linked to several "LiveJournal" blogs. The outage was recorded by pingdom at 13:31 GMT. (http://stats.pingdom.com/pdc110r2vx7j/889337/2014/10) I was notified about this outage at 13:36 GMT via SMS (Email too) by Pingdom. Which then I quickly went online and started investigating this issue.(https://twitter.com/beeimg/status/519120005695537152)



Normally whole server is monitored and if the Apache server is crashed, it automatically get restarted. during this outage the whole server got crashed making the monitoring mechanism useless. the server was returning pings, meaning the server was accessible, but was under high load. I gave the server a hard reboot at 13:50 GMT via DigitalOcean CP and the server was back online 13:52 GMT. The server was completely inaccessible for a 22 min period. (https://twitter.com/beeimg/status/519123971133169664)



To control the server load I quickly jumped to the Cloudflare CP and proxied the traffic of beeimg.com via CF for a brief time. (https://twitter.com/beeimg/status/519124680318664705) during the time I had edited the back end to redirect the images to CF powered domain. which resulted was most of the images sent to cdn.beeimg.com but gave a 404 error due to recent back end updates. I quickly applied a patch at the server end, but CF had cached the 404 error for most users for nearly 2 hours.



we were switching to Cloudflare HTTPS in the past few days, and were migrating the CDN domain to i.beeimg.com to serve the CF cached content. more test were done and new codes were added to balance the load and the issue was officially announced solved around 17:30 GMT (https://twitter.com/beeimg/status/519185562486710272)



We will be monitoring the server further more and will be adding more load balancing mechanisms during the upcoming updates to the site. Also we added a new header to moniter the server load the which will be explained in another blog article.



Thanks for reading :)
-Admin

Comments

Post a Comment

Popular posts from this blog

Stability Improvements 2020

Hi all, Since we detected that the site was operating in a degraded status causing increase rate of 500 server errors, we checked the servers and found the memory usage was high on all servers. To remedy that we upgraded the database server and rebooted all app servers clearing the memory on them and these steps reduced the error rate. Also we improved loading balancing and fail-over logic in the frontend servers. It is due to these improvements and improved thumb image caching, that we were able to release the search page to public. Earlier we constrained with less data and errors on creating thumbnail images on the results page. Please enjoy the improved service and please do reach us, if you find the service not fast enough. Thanks. Also read the stability improvements we did in 2019 in this post: https://blog.beeimg.com/2019/12/stability-improvements-2019.html
Read more

Stability Improvements 2019

Hello all, Over the past few weeks BeeIMG has been a little unstable and just wanted to clarify on what happened. In the first unstable issue which in November were caused by our server provider doing maintenance on our servers. Our AMS node got hanged up during the maintenance and had to be manually restarted. However the impact on the service was minimal and the maintenance on our CDG node affected our service as its where our databases is hosted. After the maintenance the db server had been moved to under performing hypervisor and started to slow down the site. We had to move the server again to a better server. Forward to December, one on of our free customers started hitting with lot of image view per second and this overloaded memory on AMS server and to redis node to fail. Making maters worse AMS server ran out of disk space. when a fail over to CDG happened the traffic overload that servers redis node and caused the service to fail. The site served 503 errors durin
Read more

Recent Hiccups May 2016

We faced some blackouts recently, and it was because we got very popular in china. The images that was generating lot of traffic were violating our TOS, but as they were generating lot of traffic and I thought it was a good time to stress test our image delivery systems. At first we were handling the load fine. after some time our redis server start acting wired. the issue was with dumping the db to disk, but as the images were generating lot of data the redis server crashed. we started to migrate the all gif images to be served by our CDN while still collecting views data. but at some point our real time stats handling script started to fail due to redis connection issues. after that the data started to collect and the redis server crashed every time it was started. While all that was happening we noticed that our php front end started throwing errors. this was due the connectivity issues to our redis server. The image serving end do not use redis server and was not afflicted. its usi
Read more