Stability Improvements 2019
Hello all,
Over the past few weeks BeeIMG has been a little unstable and just wanted to clarify on what happened.
In the first unstable issue which in November were caused by our server provider doing maintenance on our servers. Our AMS node got hanged up during the maintenance and had to be manually restarted. However the impact on the service was minimal and the maintenance on our CDG node affected our service as its where our databases is hosted. After the maintenance the db server had been moved to under performing hypervisor and started to slow down the site. We had to move the server again to a better server.
Forward to December, one on of our free customers started hitting with lot of image view per second and this overloaded memory on AMS server and to redis node to fail. Making maters worse AMS server ran out of disk space. when a fail over to CDG happened the traffic overload that servers redis node and caused the service to fail. The site served 503 errors during the time.
To fix this issues, a new server was added in CDG bring our server count to 3. This server will host the database and the database successfully migrated to the new server. It will also server as a storage node to free some space on CDG1.
Then we noticed that redis node getting unresponsive. when checked, the huge image request backlog created by the earlier issue made redis slow and unresponsive from time to time. We tweaked the scripts to process more requests to process the backlog.
We also moved our CDG1 server to a new one, migrating from Ubuntu to Debian.
To provide better service to every one in the future a image view limit will be added for free members
Due this instability all premium members will get 2 months added to their plan.
Thanks for reading.
Over the past few weeks BeeIMG has been a little unstable and just wanted to clarify on what happened.
In the first unstable issue which in November were caused by our server provider doing maintenance on our servers. Our AMS node got hanged up during the maintenance and had to be manually restarted. However the impact on the service was minimal and the maintenance on our CDG node affected our service as its where our databases is hosted. After the maintenance the db server had been moved to under performing hypervisor and started to slow down the site. We had to move the server again to a better server.
Forward to December, one on of our free customers started hitting with lot of image view per second and this overloaded memory on AMS server and to redis node to fail. Making maters worse AMS server ran out of disk space. when a fail over to CDG happened the traffic overload that servers redis node and caused the service to fail. The site served 503 errors during the time.
To fix this issues, a new server was added in CDG bring our server count to 3. This server will host the database and the database successfully migrated to the new server. It will also server as a storage node to free some space on CDG1.
Then we noticed that redis node getting unresponsive. when checked, the huge image request backlog created by the earlier issue made redis slow and unresponsive from time to time. We tweaked the scripts to process more requests to process the backlog.
We also moved our CDG1 server to a new one, migrating from Ubuntu to Debian.
To provide better service to every one in the future a image view limit will be added for free members
Due this instability all premium members will get 2 months added to their plan.
Thanks for reading.
Comments
Only Blog
Guest Blogger
Guest Blogging Site
Guest Blogging Website
Guest Posting Site