Some time ago I was testing how a Windows IIS powered backend would perform compared to the current Mono with Cherokee on Linux setup. The graphic shows the measured average response times. The switch to the Windows Backend was done shortly after 10PM. As you can see the average response times were significantly smaller afterwards.
Before there were 2 C3.XLarge Linux instances, afterwards there were 2 C3.Large Windows instances. So the IIS performed better in my setup on a smaller instance.
Since this setup is also cheaper than the set of Linux instances, I am thinking about a permanent switch. I am also going to do comparisons using a different Mono setup. I will test Nginx as well as Apache.
I am maintaining the backend of an online soccer app that is hosted on several Ubuntu servers in the Amazon cloud, running on Mono and ASP.NET with FastCGI on the Cherokee web server. I spent quite a while figuring out why after some time, actually only under load, the smaller servers would be stuck at 100% cpu usage consumed by the Mono process. On a medium machine this happened only once yet. In the AWS console it looked like this:
A server is running fine for a while, then suddenly jumping to 100% cpu usage and not recovering from there even though over night the load drastically decreases. It would sit there forever, still serving requests though. After restarting Mono everything would be back to normal for some time until it’s back at 100%. The peak load is around 20 requests per second.
I tried to check a complete stack dump for the Mono process but without any finds.
My latest effort was to play with some environment variable settings for Mono. Setting MONO_THREADS_PER_CPU to a high number of 2000 actually fixed the problem with the cpu going to 100%.
Since I am currently using the Cherokee web server this is an excerpt of my cherokee.conf that configures the FastCGI source:
source!1!env!MONO_ASPNET_WEBCONFIG_CACHESIZE = 2000
source!1!env!MONO_THREADS_PER_CPU = 2000
source!1!env!TERM = xterm
source!1!env!PWD = /
source!1!env!PATH = /sbin:/bin:usr/sbin:/usr/bin
source!1!env!LANG = en_US.UTF-8
source!1!env_inherited = 0
I actually had to include other environment variables that are normally inherited by mono from the web server as well (PATH, …), otherwise the application would not work.
I will continue to watch if the problem really vanished. Hope this helps somebody since it is hard to find a really good best practice guide or anything to setup and run production servers with Mono on Linux.