New Post: Mysterious Script Compile Failure after period of long running successful compile/execution

Environment:

• Windows Server 2008 R2 SP1 x64, patched up.
• .NET 4.0, patched up.
• Long-Running ASP.NET Web App "Daemons" in IIS 7.5
• * Basically, long-running apps without the suck of Windows Service applications
• Javascript.net 0.7 x64 Release
• v8 3.14.3.0 x64 Release
  

We've been using Javascript.NET for 2+ years very successfully. We use server-side configuration scripts that change rarely and that are executed several million times a day over a farm of 20+ servers.

I have one particular script that runs very, very frequently on a portion of the server farm, and it works flawlessly in multiple applications. These processes are restarted only once every 2 weeks when we deploy.

This same script runs on another "newer" portion of the server farm and in another "newer" application. This newer application runs the same identical script, but it is running much less frequently than the other applications elsewhere in the server farm. After roughly about 11 hours or so, we start to get compile errors from this script (the script code is not changing anywhere) only on the "newer" application on the "newer" portion of the server farm.

Once these compiler errors begin, they continue until the process is restarted.

Process memory on the "newer" application is stable throughout, and the application otherwise continues to function absolutely normally.

Unfortunately, we've never been able to re-create this problem in non-production environments, even with similar servers and under load tests.

The exception we get is:

Noesis.Javascript.JavascriptException
   at Noesis.Javascript.CompileScript(Local<v8::Script>* , Char* source_code, Char* resource_name) in c:\src\thirdpartyroot\src\javascript.net\branches\0.7\source\noesis.javascript\javascriptcontext.cpp:line 304
   at Noesis.Javascript.JavascriptContext.Run(String iSourceCode) in c:\src\thirdpartyroot\src\javascript.net\branches\0.7\source\noesis.javascript\javascriptcontext.cpp:line 145

The code we're using is quite simple:

    MyResponse response = new MyResponse();

    using (JavascriptContext javascriptContext = new JavascriptContext())
    {
        javascriptContext.SetParameter("output", response);

        if (null != objectInput)
        {
            javascriptContext.SetParameter("input", objectInput);
        }

        try
        {
            javascriptContext.Run(script);

            return response;
        }
        catch (JavascriptException jx)
        {
            // Log and handle the error
        }
    }

We have ensured that the app servers are identical. We have added logging to emit the script when this occurs, and the script is always intact (unchanged). Interestingly (confusingly?), if we run the "newer" application on the "older" portion of the server farm, alongside other applications, this script compile error never occurs (even over weeks).

We are currently auto-recycling our "newer" application every 11-ish hours to work-around this problem.

Has anyone run into anything similar?

Does anyone know of any v8 or Javascript.NET issues that might be related here?

Is there anyway to re-initialize Javascript.NET without an application or AppDomain restart?

Things we're investigating:

• Running an additional canary script to see if the failure is script-specific
• * Since the problematic servers are only using this one script
• Upgrading to latest GitHub code, latest stable v8
• Adding alternative support for ClearScript
• * But the performance is not nearly as good there for our use cases
• Compiling/running scripts in an isolated app domain, so we can destroy/re-create when this problem appears