Home / Archive by category "Hosting"

IIS vs. Mono/Cherokee .NET web app response times

Some time ago I was testing how a Windows IIS powered backend would perform compared to the current Mono with Cherokee on Linux setup. The graphic shows the measured average response times. The switch to the Windows Backend was done shortly after 10PM. As you can see the average response times were significantly smaller afterwards.
Before there were 2 C3.XLarge Linux instances, afterwards there were 2 C3.Large Windows instances. So the IIS performed better in my setup on a smaller instance.

Switch Linux to Windows

Since this setup is also cheaper than the set of Linux instances, I am thinking about a permanent switch. I am also going to do comparisons using a different Mono setup. I will test Nginx as well as Apache.

Mono Key Duplication Error in High Traffic Environment Workaround

After setting up the Auto Scaling feature in the Amazon cloud to automatically fire up and terminate virtual machines, depending on how much traffic there is, I ran into some issues with Mono.

In my environment I run Mono 3.4 using the Cherokee web server on Ubuntu 14.

Apparently there is a bug (that actually should be already fixed according to http://lists.ximian.com/pipermail/mono-bugs/2013-May/113692.html) which leads to erroneous Mono instances when there is high incoming traffic during startup of that instance. The bad thing is that this instance of Mono will then not recover during subsequent calls, only killing the process helps. I saw two kinds of this error, one with key duplication for key “roleManager” and one for key “sessionState”.

The stack trace of the error looks like this:

In my web.config neither role manager config entries, nor session state entires were present. But each web.config used in your own projects derives from a global web.config (just like on IIS and Windows). The location for that global web.config on Ubuntu for .NET 4 web apps is /usr/etc/mono/4.5/web.config.
Commenting role manager and session manager in the httpModules section will actually fix the problem.

So the pitfall of this workaround is that you can only use it if your web app does not use sessions and is also not using the role manager / membership functionalities.

Thanks for reading, hope this helps somebody to get Mono very stable in a high traffic production environment.

Optimizing Mono FastCGI running on Ubuntu / Cherokee Webserver

For an online game I am running Mono in the Amazon AWS cloud on several c3.large instances. Those instance types have 2 virtual CPUs. Today I experienced a little downtime of the game and found that none of the servers was working at their full capacity. Mostly only 25% CPU was used. The database servers could not be the bottleneck as the following charts showed.

Loadbalancer Request Latency
This diagram shows the average latency our load balancer in AWS encountered while serving the requests. As you can see there is a spike up to 12 seconds at around 13:45.

WebServer CPU usage
This diagram shows the CPU usage on one of our web servers (the others were similar). At around 13:45 a spike of around 40% CPU usage can be found, not nearly at the servers capacity.

Database Server CPU usage
This chart shows the CPU usage of the master database server. Also not nearly at it’s full capacity. Since we already invested quite a lot of time into optimizing the database server, I know that it is possible to max it out and found that this could not be the bottleneck.

In our current configuration the Cherokee WebServer is configured to access one Mono FastCGI source. This apparently rather inefficient. So in order to get better results I added 3 more FastCGI sources basically allowing Cherokee to load balance now between 4 FastCGI sources. I cannot yet confirm that the servers capacity is not used better, but I will update this post as soon as I can.

The cherokee configuration looks now like this:

Each FastCGI process is listening on its own port. Now in order for Cherokee to load balance between the sources, the following is needed in the configuration:

This tells Cherokee that the default handler of my default matching rule of our virtual server should use the information sources 1 to 4 and load balance between them.

Mono Running at 100% CPU Usage After Some Time (Under Load)

I am maintaining the backend of an online soccer app that is hosted on several Ubuntu servers in the Amazon cloud, running on Mono and ASP.NET with FastCGI on the Cherokee web server. I spent quite a while figuring out why after some time, actually only under load, the smaller servers would be stuck at 100% cpu usage consumed by the Mono process. On a medium machine this happened only once yet. In the AWS console it looked like this:

Before altering settings

 

A server is running fine for a while, then suddenly jumping to 100% cpu usage and not recovering from there even though over night the load drastically decreases. It would sit there forever, still serving requests though. After restarting Mono everything would be back to normal for some time until it’s back at 100%. The peak load is around 20 requests per second.

I tried to check a complete stack dump for the Mono process but without any finds.

My latest effort was to play with some environment variable settings for Mono. Setting MONO_THREADS_PER_CPU to a high number of 2000 actually fixed the problem with the cpu going to 100%.

Since I am currently using the Cherokee web server this is an excerpt of my cherokee.conf that configures the FastCGI source:

I actually had to include other environment variables that are normally inherited by mono from the web server as well (PATH, …), otherwise the application would not work.

I will continue to watch if the problem really vanished. Hope this helps somebody since it is hard to find a really good best practice guide or anything to setup and run production servers with Mono on Linux.