ASP.NET web app performance issue ThreadPool

 
One of my bank customers planned to roll out a new ASP.NET web application serving as their critical internet banking portal. They faced some weird performance issue in their UAT environment. They noticed the CPU utilization was really high (above 90%) without any fluctuation. After checking the IIS log, I found that time-taken value was really small which showed IIS can respond promptly. The stress testing was conducted in the same machine as IIS server so any suspicion on network can be diverted. As usual, I took three rounds of hang dump on IIS server as well as performance monitor log.
 
The CPU utilization on IIS server was really high and most of them was User Mode time. I also noticed the Worker Thread usage had a running number of 24. As the number shown here might not be accurate, we need to dig further on whether it was a thread pool issue.
Work Request in Queue: 0
--------------------------------------
Number of Timers: 9
--------------------------------------
CPU utilization 91%
--------------------------------------
Worker Thread: Total: 27 Running: 24 Idle: 3 MaxLimit: 200 MinLimit: 2
Completion Port Thread:Total: 2 Free: 2 MaxFree: 4 CurrentLimit: 2 MaxLimit: 200 MinLimit: 2
 
With that in mind, I dumped the System.Web.RequestQueue object and found the minExternFreeThreads value of 176 which can in turn calculate the worker threads of 24.
 
0:010> !do 06641d20
Name: System.Web.RequestQueue
MethodTable: 68a20dc8
EEClass: 68a20d58
Size: 64(0x40) bytes
GC Generation: 2
(C:\WINDOWS\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
790fed1c  4001134       14         System