Good morning AskPerf! This is a quick blog to inform you that you no longer have to contact Microsoft Technical Support to obtain the Dynamic Cache Service for Windows Server 2008 R2. It is now freely available to download from the following link:
Microsoft Windows Dynamic Cache Service
Additional Resources
-Blake
Hello AskPerf! I’m sure you already know this, but if you don’t, XP Support Ends on April 8th, 2014. That is 21 days from this post. Below are a couple of links that will give you more information on moving forward:
<SNIP from the second link above>
As a result, after April 8, 2014, technical assistance for Windows XP will no longer be available, including automatic updates that help protect your PC. Microsoft will also stop providing Microsoft Security Essentials for download on Windows XP on this date. (If you already have Microsoft Security Essentials installed, you will continue to receive antimalware signature updates for a limited time, but this does not mean that your PC will be secure because Microsoft will no longer be providing security updates to help protect your PC.)
If you continue to use Windows XP after support ends, your computer will still work but it might become more vulnerable to security risks and viruses. Also, as more software and hardware manufacturers continue to optimize for more recent versions of Windows, you can expect to encounter greater numbers of apps and devices that do not work with Windows XP.
</END SNIP>
-Blake Morrison
Good morning AskPerf! Anshuman here with a quick post on timestamp differences in Windows Explorer when accessing FTP sites.
Scenario
On Windows 7(SP1), Windows 8, and Windows 8.1, you access a folder using ftp by opening an explorer window and typing in the FQDN FTP URL in the address bar. In the FTP folder view, if you access the properties of a file, you may notice that the Modified time in the properties of the file may not match the Date modified time stamp displayed in the detailed view in explorer. See screenshot below:
This behavior can be observed if the local time set on the machine from where you access the FTP folder in explorer view is set to a Time Zone other that the UTC time. In this case you will see that the Timestamp of the item, when its property is accessed is displayed as UTC, while the timestamp that we see under the Date modified column for the same file is shown as per the Time Zone that’s set on the machine.
Please note that both the timestamps are correct. The difference is due to the Time Zone bias added or subtracted from the UTC time.
Additional Resources
-Anshuman Ghosh
Hello AskPerf Readers! Henry Chen here from the Devices & Deployment team. Today, I would like to spend some time highlighting some of the user experience changes in the shell for Windows 8.1 Update.
Windows 8.1 Update introduces numerous enhancements to the Desktop experience for mouse and keyboard users.
Start Screen
For mouse users, when right clicking anywhere on the Start Screen, a context menu will now appear in replacement of the command bar. The context menu provides the same options as what the command bar. Some of these commands are tile resizing, enable/disable live tile, and uninstall the application.
For devices with a screen larger than 8.5" and are not connected standby capable, Power and search controls can now accessible from the Start screen. For other devices, only the search control is available.
Pin Apps to Taskbar
Users can now pin modern apps to the taskbar with the exception of modern Internet Explorer. Users can block this setting by unselecting "Show Windows Store apps on the taskbar" from the Taskbar and Navigation properties.
Access taskbar from anywhere
When you are using a mouse, you can see the taskbar from any screen, including Start or a Windows Store app. Move your mouse pointer below the bottom edge of the screen to show the taskbar and then click an app to open or switch to it.
Modern App User Interface (UI)
Your mouse works more consistently anywhere in Windows. When moving the mouse to the top of the screen, the close and minimize buttons will appear from within Windows Store apps.
For devices that are touch enabled, users will continue to use the close gesture (From top edge, tap and hold dragging to bottom of the screen for a few seconds).
These are just some of the user experience changes available in Windows 8.1 Update. Let us know what you think or if you have any questions regarding these changes.
Enjoy!
For more information and to view the complete list of new features, check the following links:
What’s new in Windows 8.1 Update and Windows RT 8.1 Update?
Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2 Update April, 2014
-Henry
Hello AskPerf! Kiran Kadaba here to talk about configuring Listener Certificates.
When we have the Remote Desktop Session Host role installed on a server, or have the server as part of an RDS collection/deployment, it’s quite easy to configure certificate through the connection broker UI.
We have received a high amount of inquires on how we can configure certificates if the server is not part of a deployment, and is simply being configured for ‘Remote Desktop for Administration’.
In Windows 2003/2008/2008 R2, we had the ‘Remote Desktop Configuration Manager’ MMC snap-in which allowed us direct access to the RDP Listener. Here we could bind a certificate to the listener and in turn, enforce SSL security for the RDP sessions.
In Windows 2012, we no longer have this MMC snap-in, nor do we have direct access to the RDP listener. You can follow the below steps to configure the certificates on Windows 2012/2012 R2.
This can be achieved in 2 ways:
Method 1: Using WMI
The configuration data for the RDS Listener is stored in the ‘Win32_TSGeneralSetting’ class in WMI under the ‘Root\CimV2\TerminalServices’ namespace.
The certificate for the RDS listener is referenced through the ‘Thumbprint’ value of that certificate on a property called ‘SSLCertificateSHA1Hash’.
This thumbprint value is unique to each certificate. You can find the value using the following steps:
1. Open the properties dialog for your certificate and select the Details tab
2. Scroll down to the Thumbprint field and copy the space delimited hex string into something like Notepad
Here is what the certificate thumbprint will look like in the certificate properties:
Once I copy this into notepad, it will look as follows:
After I remove the spaces, it will still contain the invisible ASCII character that will only be visible in the command prompt (shown below):
Ensure that this ASCII character is removed before we run the command to import the certificate
3. Remove all the spaces from the string. (Keep in mind that there may be an ‘invisible’ ACSII character that also gets copied. This is not visible in Notepad. Only way to validate, would be to copy directly into the command prompt window.)
4. This is the value you need to set in WMI. It should look something like this: 1ea1fd5b25b8c327be2c4e4852263efdb4d16af4.
Now that you have the thumbprint value, here's a one-liner you can use to set the value using wmic:
wmic /namespace:\\root\cimv2\TerminalServices PATH Win32_TSGeneralSetting Set SSLCertificateSHA1Hash="THUMBPRINT"
This solution would work on Windows 7 and Windows 8 systems as well.
Note: The certificate you want to use, must be imported to the 'Personal' Certificate Store for the Machine account, before you run the above commands. Failure to do so will result in a “Invalid Parameter” error.
Option 2: Registry edits
3. The Remote Desktop Host Services service runs under the NETWORK SERVICE account. Therefore, it is necessary to set the ACL of the key file used by RDS (referenced by the certificate named in the SSLCertificateSHA1Hash registry value) to include NETWORK SERVICE with "Read" permissions. To modify the permissions follow the steps below:
Open the Certificates snap-in for the local computer:
-Kiran
Hello AskPerf! My name is Matt Graham and I will be writing a high level overview of the capabilities of Windows Performance Monitor. The intention of this blog post is to introduce new users to this powerful, and often underutilized, tool. So rather than going through each part of Performance Monitor and explaining it in depth, my aim here is to offer a quick guide to the tool.
When you first open Performance Monitor (perfmon), you see the following:
Let's briefly go through each one and talk about what they do.
Performance
At the very top level "Performance" gives you an overview of your systems memory usage, network usage, disk usage, etc. You can right click on "Performance" and connect to another computer to view a remote computers performance statistics. (NOTE: Should add brief comments about what is required in order to remotely connect to another machine…)
Monitoring Tools
From the Monitoring Tools icon you can right click and launch the Resource Monitor. Resource Monitor is another powerful tool that can help you see how your system resources are being used. You also have the ability to launch the System Reliability Monitor. This utility allows you to see information about software updates and installations. You can also see critical events that occurred and on what day those events occurred. Finally, you can see all of the problem reports that have been sent from your computer by clicking on the "View all problem reports" link at the bottom of the window.
Performance Monitor
The Performance Monitor is primarily for viewing real time statistics. By default only one counter is selected; the %Processor Time counter. However you can add additional counters by clicking on the green plus sign. This will allow you to monitor any counters you wish in real time.
While you can see all of the performance counters you like here, the real power of Performance Monitor is found in its ability to capture performance metrics over an elapsed period of time. Capturing data over a period of time allows you to see trends and these trends are what are most useful for determining the overall performance of your system. To capture this data, you can create what are called "Data Collector Sets".
Data Collector Sets
Data Collector Sets are aptly named. They collect data from your system so that you can view changes in configuration information and performance information over a specified period of time.
There are basically three types of data collector sets:
Performance Counter | Capture data based on the polling of an event at a specified time interval |
Event Traces | Capture data based on events that occur rather than based on a specified time interval |
System Configuration Information | Capture configuration information |
Under Data Collector Sets you can see the following:
User Defined
Under User Defined, you can create your own custom Data Collector Set. These Data Collections Sets can contain counters, traces, and configuration collectors. You can right click on User Defined and select New -> New Data Collector Set to create one. You can create a data collector set from a template or create your own custom set. Let's create a custom one:
1. Name your Data Collector Set and select "Create manually (Advanced)".
2. On this screen you can choose to create data logs (counter / trace / config) or you can create a Performance Counter Alert. The "Performance Counter Alert" option allows you to create alerts based off of certain performance values and thresholds. For now, we will select "Create Data Logs" and place a check box in all three boxes:
3. On the screen below you can set the counter interval rate (how often do you want it to capture the selected data) and the specific counters that you want to capture.
4. Once you click Add, you can select counters and then add them to the "Added Counters" box. Note that you have options in terms of whether you want Perfmon to collect the data as a total or if you want to break the data up, in this case, per processor. You should pay attention to which one you select as the meaning of these counters depends on what is being counted.
5. You will then be prompted to add trace providers. Trace providers simply provide information to perfmon about a specific set of events. For example, if you wanted to collect event information about the Windows Firewall, you would select the "Microsoft-Windows-Firewall" provider. You can then edit the properties (if you know what you are doing) and even record registry keys (after you hit next you can specify which keys to record).
6. You can also specify a location where you would like to save the data. By default, the data is saved to: %systemdrive%\PerfLogs\Admin\New Data Collector Set.
7. Finally, you can select a user that you would like to run the data collector set as. This may be helpful in environments where desktops and servers are locked down for security purposes.
If you look at the "New Data Collector Set" that we just created, you can see that it contains a performance counter, a trace, and a configuration collector. You can right click on any of these to modify them as you see fit.
Finally, if you look at the remaining items under Data Collector Sets, you can see a bunch of preconfigured collector sets.
Reports
The final part of Performance Monitor is the Reports section. Here you can view the information that was collected by your data collector sets. If you have never run your data collector set, then you will not see any information when you click on it.
However, once you have run your data collector set, you can click on it and see the reports and information collected:
So this is a basic overview of Windows Performance Monitor. Once you are familiar with the parts, you can then dive into learning about which counters to use when, and what your counter interval rate should be when you are trying to capture this kind of data vs that kind of data.
Additional Resources
Hello AskPerf! My name is Jeffrey Worline and I wanted to let you know that I will be writing several troubleshooting WMI blog posts over the next few days. These will not be your typical Blog post, but more around the KB/Technet format.
There has been a lot of internal discussions here at Microsoft with Product Group, Developers, and myself concerning the high amount of WMI issues that have come into Support. Whenever there is an issue with WMI itself, or suspected issue with WMI, the first words that always seem to come out is that “WMI is corrupted”. Oh no, say it isn’t so. What do we do? Grab the “Big Hammer” and rebuild the repository! Excuse my humor as we have a little fun, as the upcoming blogs are going to be more straight forward and dry.
While rebuilding the repository may seem like a good idea and a quick fix at times, what most do not realize is that if you suspect WMI or repository corruption, rebuilding repository is the last thing you should do without verifying this is truly the case. Deleting and rebuilding the repository can cause damage to Windows and/or to your installed applications. That last sentence is worth repeating; “Deleting and rebuilding the repository can cause damage to Windows and/or to your installed applications.” When it becomes necessary to rebuild the repository and this has been verified, there are right and wrong ways to go about it.
Other steps should be taken first to eliminate other possibilities or to confirm we have repository corruption before taking such actions as rebuilding the repository. There are also other proactive steps that can be taken as alternatives to rebuilding the repository if such an occasion arises.
So, the following blog posts will address various scenarios surrounding WMI, how to troubleshoot them, and the data we should collect to help resolve WMI related issues. These posts will equip you with the information you need to address WMI issues in the correct manner, and to help prevent you from causing more damage by just using the proverbial “Big Hammer” approach.
With that, here is a list of the Posts/Topics you will see in the coming days:
-Jeff
Quota Violation Issues: Memory and/or Handle
Symptoms
Event Source: Service Control Manager
Event ID: 7031
Description: The Windows Management Instrumentation service terminated unexpectedly
Source: Microsoft-Windows-WMI
Event 5612 Wmiprvse.exe exceeding handle quota limit Event
WMI has stopped WMIPRVSE.EXE because a quota reached a warning value. Quota: %1 Value: %2 Maximum value: %3 WMIPRVSE PID: %4
Memory Quota Violations refer to Ask Perf blog: WMI: High Memory Usage by WMI Service or Wmiprvse.exe
Handle Quota Violations refer to Ask Perf blog: WMI:How to Troubleshoot WMI High Handle Count
Missing or Failing WMI Providers or Invalid WMI Class
Symptoms
Failed to initialize all required WMI classes
Win32_processor: WMI: Invalid namespace
Win32_WMISetting: WMI: Invalid namespace
Security information: Successful
Win32_OperatingSystem: WMI: Invalid namespace
Win32_processor: WMI: Invalid namespace
Win32_WMISetting: WMI: Invalid namespace
Win32_OperatingSystem: WMI: Invalid namespace
Refer to Ask Perf blog: WMI: Missing or Failing WMI Providers or Invalid WMI Class
High Memory Usage by WMI Service or Wmiprvse.exe
Symptoms
Refer to Ask Perf blog:WMI: High Memory Usage by WMI Service or Wmiprvse.exe
High CPU Usage by WMI Components
Symptoms
Refer to Ask Perf blog: WMI: How to troubleshoot High CPU Usage by WMI Components
High Handle Count on WMI Components
Symptoms
Source: Microsoft-Windows-WMI
Event 5612 Wmiprvse.exe exceeding handle quota limit Event
WMI has stopped WMIPRVSE.EXE because a quota reached a warning value. Quota: %1 Value: %2 Maximum value: %3 WMIPRVSE PID: %4
The User Profile Service service failed the logon
Refer to Ask Perf blog: WMI: How to Troubleshoot WMI High Handle Count
Repository Corruption
Symptoms
Refer to Ask Perf blog: WMI: Repository Corruption, or Not?
Next up: WMI: Repository Corruption, or Not?
Scenario
Windows Management Instrumentation failing due to repository being corrupted
The WMI Repository (%windir%System32\Wbem\Repository) is the database that stores meta-information and definitions for WMI classes; in some cases the repository also stores static class data as well. If Repository becomes corrupted, then the WMIservice will not be able to function correctly.
Before grabbing that preverbal hammer approach and just rebuilding your repository, ask yourself, “Is the WMI repository OK?”
Common symptoms that lead to this question are: provider load failure, access denied, class not found, invalid namespace, and namespace not found to mention a few.
If you suspect WMI or repository corruption, rebuilding repository is the last thing you should do without verifying this is truly the case. Deleting and rebuilding the repository can cause damage to Windows or to installed applications. Other steps should be taken first to eliminate other possibilities or to confirm we have repository corruption. Noting here that having a repository too large also creates problems; an issue that can sometimes be interpreted as a corrupt repository, which is not always the case. If issues are due to a large repository, rebuilding the repository is currently the only method available to reduce it back to a working size.
Since I mentioned “large repository”, let me set some guidelines up front. There is no hard fast number per say as to when you will start feeling performance problems with a large repository. As a guideline, if the objects.data file, located in (%windir%System32\Wbem\Repository,is 1 GB or larger, then I would recommend rebuilding your repository to reduce it back down to a working and manageable size. If the size is between 600-900 MB, and you are not feeling any noticeable performance issues, then I would recommend against rebuilding the repository.
If WMIis corrupted, you can receive various errors and symptoms, depending on what activity was being done at the time. Below is a few errors and symptoms that could indicate that the repository is corrupted:
get-cimclass : Unable to complete the requested operation because of either a catastrophic media failure or a data structure corruption on the disk.
At line:1 char:1
+ get-cimclass -Namespace root\cimv2\TerminalServices
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Get-CimClass], CimException
+ FullyQualifiedErrorId : HRESULT 0x8007054e,Microsoft.Management.Infrastructure.CimCmdlets.GetCimClassCommand
Check the Windows Application log for events in the past week where Source = Microsoft-Windows-WMI. Look for any of the following WMI event IDs: 28, 65, 5600, 5601, 5614. Any of these could indicate a WMI repository issue or core infrastructure problem.
If you do not find any of these events logged, your next action is to use the built in repository checker. From an elevated command prompt run "winmgmt /verifyrepository". If the repository has an issue, it will respond "repository is not consistent".
If repository check comes back as “consistent”, then look at my other Ask Perf blogs for applicability:
WMI: Missing or Failing WMI Providers or Invalid WMI Class
WMI: High Memory Usage by WMI Service or Wmiprvse.exe
How to troubleshoot High CPU Usage by WMI Components
WMI Self-Recover
When the WMI service restarts or detects Repository corruption, the self-recovery procedure will trigger automatically in two approaches (one or the other):
Note: Under almost no circumstance should you use the script that rebuilds the WMI repository from the MOF files
The script is inherently flawed, for 2 reasons:
In addition to causing damage to your system that's almost impossible to fix correctly, if you take that approach you will blow away all information that could be used to determine the root-cause.
If the repository check (winmgmt /verifyrepository) comes back as inconsistent, your first action is to run “winmmgmt /salvagerepository” followed by running “winmgmt /verifyrepository”again to see if it now comes back as consistent.
If it is still comes back inconsistent, then you need to run “winmgmt /resetrepository.” Before running this, please read the important note below for Server 2012.
Force-recovery process-- rebuild based on the registry list of AutoRecover Mofs
net stop winmgmt /y
then run
Winmgmt /resetrepositoy
WMI repository reset failed
Error code: 0x8007041B
Facility: Win32
Description: A stop control has been sent to a service that other running services are dependent on
NOTE: Applies to Server 2012
We have encountered some issues when running the mofcomp command on Windows Server 2012 which has caused the Cluster namespace to be removed due to the cluswmiuninstall.mofcontained in the c:\windows\system32\wbem folder. It has also caused unregistering class names for Cluster Aware Updating (CAU) because of its cauwmiv2.mof also in the wbem folder. Those could also affect other namespace that have an uninstall type .mof in the wbem folder beyond the two mentioned above.
Furthermore, the uninstall .mof files for servers running Microsoft Cluster are also part of the autorecover folder that is used when you run the winmgmt /resetrepositorycommand which will end up having the same effect of first installing the Cluster namespace, then uninstalling it just as if you had run a script to rebuild the repository that contained the “for” command to recompile all of the MOFs in the WBEM folder.
Take the following actions to confirm if the uninstall problem for this scenario exists on your server. If it doesn’t, then you can run the winmgmt /reset repository, otherwise follow my directions below for manually accomplishing rebuild.
How to manually rebuild repository on Server 2012 Cluster machine when cluster provider uninstall has an autorecover
First ensure you have run winmgmt /verifyrepository to ensure that it is “inconsistent” and that you have tried winmgmt /salvagerepositoryto see if it resolves your issue.
Change the startup type for the Window Management Instrumentation (WMI) Service to disabled.
As a final note, if you run into a reoccurring corruption issue in your environment with WMI, try to exclude the WBEM folder and all subfolders under it from AV scanning. AV scanning is known to cause corruption and other issues in WMI.
Other repository recovery solutions:
Note: in following solutions (1 & 2), if the backup images (repository) are in large size (>100MB), restoring the repository will take some time.
$filterStmt = [String]::Format("DriveLetter='{0}'",$env:SystemDrive)
# Get systemdrive Volume info
$vlm = gcim win32_Volume -Filter $filterStmt
# create a shadowcopy
$res = icim win32_ShadowCopy -MethodName Create -Arguments @{Volume=$Vlm.DeviceID}
if ($res.ReturnValue -eq 0)
{ gcim win32_ShadowCopy -Filter ("ID='"+$res.ShadowID+"'") } # **
else
{ $res | fc }
NoteThe WMI Service has auto start setting, and if it comes back alive, you will not be able to replace the files. The service needs to be in a stopped state (if WMI service is alive at the time, repeat step: 2~3)
ex. Directory of \\localhost\C$\@GMT-2014.03.13-01.02.49Windows\system32\wbem\repository
03/11/2014 11:53 AM <DIR> .
03/11/2014 11:53 AM <DIR> ..
03/12/2014 05:30 PM 4,759,552 INDEX.BTR
03/12/2014 05:30 PM 90,640 MAPPING1.MAP
03/12/2014 03:26 PM 90,640 MAPPING2.MAP
03/12/2014 05:24 PM 90,640 MAPPING3.MAP
03/12/2014 05:30 PM 27,541,504 OBJECTS.DATA
4. Run following wmic command to bring WMI service back to life: wmic os
You should set up a regular Scheduled Task to backup the latest repository:
You could also schedule restores as necessary
If the issue is not a repository issue, and the objects are not retrievable:
If the issue persists or keeps returning, then at this point you will now need to open a Support Incident Case with Microsoft for further assistance.
Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and well help us track the effectiveness of the blog.
Next up: WMI: Missing or Failing WMI Providers or Invalid WMI Class
Scenario
Windows Management Instrumentation fails due to receiving an event or error concerning missing or failure to load WMI Provider, or Invalid WMI class, or WMI Invalid Namespace.
Below are some common errors indicating issues with a WMI Provider or Class:
The common urge by many is to simply take a hammer to WMI and either rebuild the repository or recompile all of the .mof files in the C:\Windows\System32\Wbemfolder. While the hammer may seem a good approach to solve your problem in some cases or what seems like a quick fix, it can actually cause more problems and issues that are not immediately visible or felt. These newly created problems as a result of the hammer approach are not always immediately obvious and tend to surface later, and usually when the last thing you need is another problem to have to deal with on your plate, especially if it created a critical business situation.
Rebuilding the repository or recompiling all of the .mof files as a first action when other steps should be taken first can cause damage to the system and/or to installed applications.
First we want to just check right up front if our repository is corrupted or not to save time, then continue from there with appropriate actions.
Check the Windows Application log, look for events in the past week where Source = Microsoft-Windows-WMI, check if any of the following WMI event IDs exist: 28, 65, 5600, 5601, 5614. Any of these could indicate a WMI repository or core infrastructure problem.
Check if the repository is corrupted or not by running following command from command prompt with admin rights: winmgmt /verifyrepository. If it is corrupted the returned result will be “Repository is inconsistent.” If it comes back as consistent, then repository is not corrupted so do not rebuild repository at this point.
Note: if results came back as “inconsistent”, then skip to bottom of the page and reference the section titled “Wmi Repository Corrupted”, otherwise continue with the methods below.
If issue is with a 3rd party provider or namespace, the vendor should be engaged as they are responsible for their own namespaces and providers.
For the 3 scenarios listed below for WMI Invalid Namespace, WMI Invalid Class, and WMI Provider Load failure, if a class is present and operation still errors out with invalid class, then the most likely reason is that service/wmiprvse is hitting memory quota limit or issues.
When a quota issue is hit in wmiprvse, new memory allocations fail and based on where the failure happen client gets different errors “invalid query”/Invalid class/Provider load failure/quota check”
When a quota issue is hit when a new operation is for a provider which is not loaded already in wmiprvse and thus requiring it to be loaded in a new instance of wmiprvse, then the provider loading would fail with “provider load failure”.
Scenario 1: WMI Invalid Namespace
First we want to take any scripts or programs out of the equation by using local built in tools. The two most common tools used to check wmi functionality is the WMI console (winmgmt.msc) and Wbemtest (Windows Management Instrumentation Tester).
Ensure the Namespace in question actually exist and functional.
a. Open up a command prompt with administrative rights or elevated privileges
b. Go to www.msdn.com and search for the provider of the missing namespace. We are looking for the associated .mof file. In the below example, we will use MSCluster as our missing namespace.
e.g. search words: wmi mscluster provider or wmi mscluster namespace. In this case you should find ClusWmi.mof as the WMI provider
In case the CimV2 namespace is missing, the provider is CimWin32.mof
If it also lists the associated .dll file, you will want to re-register it also as this would be the registration for the DCOM side as WMI and DCOM work hand in hand.
c. From the command prompt with administrative rights or elevated privileges change directory to C:\Windows\System32\Wbem and run the following command: mofcomp.exe <provider.mof>
Note: Replace <provider.mof> with actual .mof name you found searching MSDN. In our example above that command would be: mofcomp.exe ClusWmi.mof
For re-registering associated .dll if one exists use following command: regsvr32 <provider.dll> and again replacing <provider.dl> with actual dll name
d. Restart the Windows Management Instrumentation Service
e. Use wmimgmt.msc console again to now check if the missing namespace is now listed. First close the console and reopen if it was still open from previous action
7. Go to start-run and type in wbemtest
8. Click on the “Connect Button”
9. In the Namespace Box type in the path to the namespace for which getting invalid namespace error for. This path would have the same look and feel of a Windows Directory, so just as you see the structure in wmimgmt.msc console on the Securitytab, so is how you will type in path
Examples:
Root\Cimv2
Root\Mscluster
Root\RSOP\Computer
10. Click on the “Connect” button
11. Now all of the buttons should no longer be greyed out on the main wbemtest console page. Click on the “Enum Classes” button
12. Leave “Enter Superclass Name” blank and select “Recursive” then click OK. If you don’t get any error messages then you can access the name successfully without issue using built in Windows Management Instrumentation Tester
13. To test further, let’s see if we can access some classes.
a. Pick any class and double click on it to bring up the Object Editor for that class
b. Next click on the “Instances” button on the right
c. If it doesn’t sit there hung at “Operation in progress” or doesn’t return any error, then access to that class would appear to be okay.
NOTE: Not every class you click in the “Instances” button will actually return results. Some classes will and some won’t depending on the class. Don’t sweat it if it doesn’t as long as the box above says “Done”
Repeat this on several classes in the namespace for sanity check just to see if any produce an error message or Operation in progresshangs.
If you did not get any errors connecting to the namespace or accessing some of the classes in that namespace, then your issue may be with the application or whatever method being used outside of Wbemtest that is the problem and would advise opening case with Microsoft if it is a Microsoft application or process, otherwise first contact the vendor of the application for assistance in troubleshooting.
Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.
Scenario 2: WMI Invalid Class
To troubleshoot an Invalid WMI Class, you basically would follow the same procedure as above that you would for WMI Invalid Namespace.
Here is a good link that you could probably also click your way thru various links to find the provider .mof for Microsoft classes.
Scenario 3: WMI Provider Load Failure
Again, you will follow the same procedures as above for testing, and also locating the provider .mof file and associated .dll if one is listed and recompiling and re-registering of that particular provider.
Wmi Repository Corrupted
There are some exceptions where you have accomplished all of the above and from Wbemtest (note I said Wbemtest, not some method outside of Wbemtest) and you are still getting failures with a namespace, class, or provider. In these scenarios you may have to revert to using the proverbial hammer as a last ditch effort even though the repository check came back as consistent when your ran winmgmt /verifyrepository.
Before doing such, it would be a good idea to open a Support Incident Case with Microsoft for assistance before reverting to rebuilding of the repository if the winmgmt /verifyrepository had come back as Consistent.
IMPORTANT
If this becomes the case, please refer to my Ask the Performance Team Blog article, WMI: Rebuilding the WMI Repository before rebuilding the repository as there are some important gotchas that you need to be aware of.
As a final note, if you are running into a reoccurring WMI corruption issue in your environment, try to exclude the WBEM folder and all subfolders under it from AV scanning. AV scanning is known to cause corruption in WMI.
At this point you may also want to open a Support Incident with Microsoft for further assistance if needed.
Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.
Next up: WMI: High Memory Usage by WMI Service or Wmiprvse.exe
Scenario
Windows Management Instrumentation Service (Winmgmt) or WMIprovider (wmiprvse.exe) is consuming high amounts of memory.
There are many reasons why WMI might experience high memory consumptions. This can occur in the WMI (Windows Management Instrumentation) service (winmgmt) or in the WMI provider hosting vehicle wmiprvse.exe. Both scenarios will be addressed below.
High memory usage may simply be due to load, as opposed to some type of leak. By default, the memory quota limit for instances of wmiprvse.exe on Windows XP and Windows Server 2003 is 128 MB, and 512 MB on newer Operating Systems (Vista and higher). Hit those limits and wmi functionality will become problematic if not come to a grinding halt.
There is another case where the quota limit problem could happen. There is limit for all wmiprvse’s cumulatively. If the total memory of all instances of wmiprvse’s (with one exception) together reaches 1GB, then all new memory allocations fail in all wmiprvse processes.
As a first effort you can try to bump up that quota limit to see if it gives enough room for wmiprvse to operate, but if you still reach the new quota limit, then you will want to proceed with the Scenarios below
How To Bump up WMI memory quota limit:
Note: you aren’t connecting to CimV2 or any other namespaces. It’s ROOT
Note: if you cannot restart the machine, then as a workaround, you can break Windows Management Instrumentation service into its own svchost process outline in step 2. a-c below.
If bumping the quota limits does not resolve the issue, then as workaround you can try to move suspected leaking providers into their own group (wmiprvse) to avoid the memory quota caused by other providers running in the group or kill the instance of wmiprvse exhibiting high memory until issue is resolved. This is outlined below in Scenarios 1a and 2a.
First thing we need to determine is if the memory consumption being caused by private data or heap data. We have to address the 2 types differently.
1. Download a Windows Sysinternals tool call VMMap from following url: http://technet.microsoft.com/en-us/sysinternals/dd535533.aspx
VMMapis a process virtual and physical memory analysis utility. It shows a breakdown of a process's committed virtual memory types as well as the amount of physical memory (working set) assigned by the operating system to those types.
This tool is used to attach to an individual process allowing a snapshot to be taken to see the memory map for that process.
2. Simply launch VMMap and from the process list it displays, pick the instance of wmiprvse showing the high working set memory usage.
3. If it is a svchost process that is exhibiting high memory, from a command run scqueryex winmgmt to identify the PID of the svchost process that is hosting WMI Service (winmgmt). From experience it will be the WMI service more times than not but not always as the service using majority of the memory; as such I would try to break it out first on its own and monitor to see if it is the one driving up high memory usage in the shared svchost process.
From this point, I will assume it is the WMI service as this article is only addressing WMI and no other services.
You will need to break the WMI service out into its own svchost process first if it hasn’t already been accomplished so you can run VMMap specifically against the WMI service to truly know if it is being consumed by Heap or Private Data. By default the WMI service runs in a shared svchost process with a community of other services. You can do so by the following directions. If it is caused by Heap, you will find later in the Scenario directions below that you will have to go back and uniquely name the svchost process for the service to run in to be able to enable User Stack Tracingagainst it.
a. Open command prompted with elevated privileges
b. Break wmi service out into its own svchost process by running following command: sc config winmgmt type= own
c. Restart wmi service with net stop winmgmt and net start winmgmt commands
d. Verify winmgmt service running in its own svchost process by runnning tasklist /svc
4. Once it displays the result, look under the size column for Private Data and Heap. This should tell you if the majority of the memory being consumed is Private Data or Heap.
5. Follow appropriate Scenario below based on if issue is with Windows Management Instrumentation service or wmiprvse and if caused by Heap or Private Data memory.
Scenario 1a: High Memory Consumption by Wmiprvse.exe caused by Heap memory
1. Download Windows Debugging Tools for Windows from following link: http://msdn.microsoft.com/en-us/windows/hh852365.aspx
You can probably just get the standalone version since we only need the debugging tool and not the whole WDK package.
2. Go to the directory where you installed the tool and you will find gflags.exe as one of the files, right click on it and select run as administrator
3. Click on “Image File” tab
4. Type in wmiprvse.exe
5. Hit the keyboard TAB key
6. Place check mark in “Create user mode stack trace database”
7. Click Ok
8. Download the latest version of the Windows Sysinternals tool Process Explorer. http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx
9. Find the instance of wmiprvse.exe with high memory consumption and right click on it and bring up the properties sheet. Click on the WMI Providers tab and document the listed providers
Alternatively you could run following wmiccommand from command prompt to get a list of loaded providers and their associated PID you could then match up to PID with instance of wmiprvse with high memory usage:
wmic path msft_providers get Provider,HostProcessIdentifier /format:list
10. We now want to take that list of providers and configure them to run in their own instance of wmiprivse.exe. Open an instance of PowerShell with admin rights
11. Run the following commands: Replace ProviderName below with actual ProviderNamer found from step 9, retain the quote marks.
$prv = gcim -namespace root/standardcimv2 __win32provider -filter "name='ProviderName'"
<ENTER>
$prv.HostingModel = $Prv.HostingModel + ":OWN"
<ENTER>
set-ciminstance -inputobject $prv
<ENTER>
Repeat the above for each Provider found in step 9
12. Restarting of Wmi service will put the settings into effect without having to reboot machine
13. Now you will want to take a process dump of the instance of wmiprvse when memory usage is high. Download Windows Sysinternals tool called Procdump from URL: http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx
14. Open a command prompt with elevated or administrative rights and change to the directory where you saved Procdump
15. Open Task Manager and add the PID column view then go locate the instance of wmiprvse.exe with high memory usage and note the PID, or use Process Explorer
16. Run the following command: procdump –ma -s 300 -n 3 <PID>
Note: Replace <PID> with actual PIDyou documented for instance of wmiprvse.exe exhibiting high memory usage
The above command will produce 3 dumps spaced at 5 minutes apart each in same directory you ran the procdump command from
17. Note also as workaround until issue gets resolved you could simply kill the instance of wmiprvse with high memory from task manager or from PowerShell can run following:
kill -f <pid of suspect wmiprvse>
At this point you will now need to open a Support Incident Case with Microsoft to get the data analyzed to determine cause of high memory usage
Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.
Scenario 1b: High Memory Consumption by WMI (Windows Management Instrumentation) service caused by Heap memory
The Windows Management Instrumentation service runs under the display name of winmgmtand is located in networking svchost.exe process shared along with several other services. You need to break the WMI service out into its own unique svchost process for data collection purposes.
NOTE: If you have already broken the WMI service out to run in its own general svchost process following directions at beginning of this article, then you can skip to step 6 to uniquely name the svchost process it is running in. We do this because we only want to enable User Stack Tracingagainst just the WMI service and not every svchost process.
Note: It is important that you go back and reverse what you did in step 7 and modify path back to original after you are no longer needing the service to be broken out and uniquely named as failure to do this can prevent future WMI hotfixes from being installed.
Simply run following command then restart wmi service: sc config winmgmt type= share
8. Restart wmi service with net stop winmgmt and net start winmgmt commands again
9. Verify you now see wmisvchost.exe process running by running tasklist or looking in task manager at process list
10. For sake of brevity here, follow steps 4-14 from Scenario 1a with the exception of typing in wmisvchost.exe in place or wmiprvse.exe for process we are enabling “Create user mode stack trace database” against.
11. Here we want to use procdump to dump the wmisvchost.exe process along with every instance of wmiprvse.exe that is running. Reference steps 9-11 listed in Scenario 1a above.
First command will dump wmisvchost.exe 3 times spaced at 5 minute interval. Once it has completed, and then run second command to dump out each instance of wmiprvse.exe that is running.
Command for dumping wmisvchost.exe would be: procdump –ma -s 300 -n 3 wmisvchost.exe
Command for dumping each instance of wmiprvse just once: procdump –ma <PID>
Note: Replace <PID> with actual PID for each instance of wmiprvse.exe
At this point you will now need to open a Support Incident Case with Microsoft to get the process dumps analyzed to determine cause of high memory usage
Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.
Scenario 2a: High Memory Consumption by Wmiprvse.exe caused by Private Data memory
This scenario is a little bit different than being caused by Heap, as just dumps in of themselves normally do not tell the full story. In this type of case, we need to know what type of wmi activity is occurring and it becomes significantly harder and more involved to try and determine cause.
We will need to collect procdumps still, but also need to add in WMI activity logging.
1. From start run type in eventvwr.msc
2. On the View menu at the top select “show analytical and debug logs” so it displays check mark next to it
3. Expand Applications and Services\Microsoft\Windows\WMI-Activity
Right click on items Debug and Trace and select “Enable” for each one
4. Download the latest version of the Windows Sysinternals tool Process Explorer http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx.
5. Find the instance of wmiprvse.exe with high memory consumption and right click on it and bring up the properties sheet. Click on the WMI Providers tab and document the listed providers
Alternatively you could run following wmiccommand from command prompt to get a list of loaded providers and their associated PID you could then match up to PID with instance of wmiprvse with high
memory usage:
wmic path msft_providers get Provider,HostProcessIdentifier /format:list
6. We now want to take that list of providers and configure them to run in their own instance of wmiprivse.exe. Open an instance of PowerShell with admin rights
7. Run the following commands: Replace ProviderName below with actual ProviderNamer found from step 9, retain the quote marks.
$prv = gcim -namespace root/standardcimv2 __win32provider -filter "name='ProviderName'"
<ENTER>
$prv.HostingModel = $Prv.HostingModel + ":OWN"
<ENTER>
set-ciminstance -inputobject $prv
<ENTER>
Repeat the above for each Provider found in step 5
8. Restarting of Wmi service will put the settings into effect without having to reboot machine
9. When memory usage is high, use Procdump tool to dump out the wmiprvse.exe exhibiting high memory usage. http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx
10. Open a command prompt with elevated or administrative rights and change to the directory where you saved Procdump
11. Open Task Manager and add the PID column view then go locate the instance of wmiprvse.exe with high memory usage and note the PID, or use Process Explorer
12. Run the following command: procdump –ma -s 300 -n 3 <PID>
Note: Replace <PID> with actual PIDyou documented for instance of wmiprvse.exe exhibiting high memory usage
The above command will produce 3 dumps space out 5 minutes apart each in same directory you ran the procdump command from
13. Collect a network trace for 2 minutes using Network Monitor. Latest version can be found at: http://www.microsoft.com/en-us/download/details.aspx?id=4865
14. Now save wmi-activity traces in Event Viewer, do so by right clicking on each and select “Save all events as” option
At this point you will now need to open a Support Incident Case with Microsoft for data analysis and most likely for more in depth troubleshooting and data collection methods
Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.
Scenario 2b: High Memory Consumption by WMI (Windows Management Instrumentation) service caused by Private Data memory
NOTE: If you have already broken the WMI service out to run in its own general svchost process following directions at beginning of this article, then you can skip to step 5
Right click on items Debug and Trace and select “Enable” for each one
8. Here we want to use procdump to dump the svchost process housing the WMI Service along with every instance of wmiprvse.exe that is running. Reference steps 9-10 listed in Scenario 1a above
a. From command prompt run tasklist /svc and locate the instance of svchost that winmgmt is running in and note the PID
b. First command will dump svchost.exe 3 times spaced at 5 minute interval. Once it has completed, and then run second command to dump out each instance of wmiprvse.exe that is running just once
Command for dumping svchost.exe housing winmgmt would be: procdump –ma -s 300 -n 3 <PID>
Note: Replace <PID> with actual PID for svchost process housing winmgmt
Command for dumping each instance of wmiprvse running would be: procdump –ma <PID>
Note: Replace <PID> with actual PID for each instance of wmiprvse.exe
At this point you will now need to open a Support Incident Case with Microsoft for data analysis and most likely for more in depth troubleshooting and data collection methods
Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.
Next up: WMI: How to troubleshoot High CPU Usage by WMI Components
Scenario
Windows Management Instrumentation Service (Winmgmt) or WMI provider (wmiprvse.exe) is consuming high amounts of CPU.
In the directions below, you may have already broken out WMI Service to troubleshoot your issue. By default, WMI runs in the main shared networking svchost process with several other services.
If it is a svchost process showing high cpu usage, you can use Task Manager and add PID column, then identify which svchost process has the high memory usage. From inside a command prompt you can type in tasklist /svc and look for the PID #, and identify if a single service is running in that svchost process or multiple services. If multiple services, it may become necessary to break each service out to run in its own svchost process to determine if it is the WMI service (winmgmt) that is causing the issue. From my experience, it will be the WMI service more times than not but not always. As such, I would suggest breaking it out first into its own, and monitor to see if it is the one driving up high memory usage in the shared svchost process.
If you suspect the WMI (Windows Management Instrumentation) service, you can break it out following directions below.
Break WMI Service out into its own svchost process
When issue had been resolved or no longer needing the service broken out into its own svchost process, place it back into the shared svchost process by running following command from command prompt:
Configure Perfmon Collection using logman.exe method. Capture 15 minutes while issue is occurring.
Short, high resolution log – 1 sec interval with thread counter, 250MB
1. Click on Start
<<Start Search>>, enter "CMD.exe" w/o the quotation marks and then press Enter.
2. Copy and paste the following command into the command prompt window (if this does not work, you may need to manually type it in):
Logman.exe create counter PerfLog-Short -o "c:\perflogs\PerfLog-Short" -f bincirc -v mmddhhmm -max 250 -c "\Cache\*" "\LogicalDisk(*)\*" "\Memory\*" "\Network Interface(*)\*" "\Paging File(*)\*" "\PhysicalDisk(*)\*" "\Processor(*)\*" "\Process(*)\*" "\Redirector\*" "\Server\*" "\System\*" "\Server Work Queues\*" "\Thread(*)\*" -si 00:00:01
3. Start the log with:
Logman.exe start PerfLog-Short
4. Please stop the performance log as soon as the issue returns with the following command:
Logman.exe stop PerfLog-Short
Please note that if you reboot the server, you will need to start the logs again as they will not automatically restart on boot.
Collect and Xperf trace for High CPU by using the Windows Performance Recorder form the Windows Performance Toolkit which you can install from the ADK
Note: If the Operating System is a 64 bit box, you must first accomplish the following registry setting before collecting Xperf trace.
Registry Path
HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management
Setting
DisablePagingExecutive
Data Type:
REG_DWORD
Value:
1
NOTE setting this key is not needed on Windows Server 2012 & 2012 R2
Reboot machine to place registry setting into effect.
After installation has finished, start creating a trace by starting the "Windows Performance Recorder"
Select CPU usage under Resource Analysis
Logging mode can be left set to “Memory”, or you can change to “File”. Just be conscious of your disk space if you chose “File” as the etl file can become large fast
Capture high cpu occurrence, but do not let the recording run for no more than 10 minutes.
Immediately after capturing the event using Windows Performance Recorder (WPR), now use process explorer to dump out the process exhibiting high cpu usage.
1. Download Windows Sysinternals tool called Procdump: http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx
2. Open a command prompt with elevated or administrative rights and change to the directory were you saved Procdump
3. Open Task Manager and add the PID column view, then go locate the instance of wmiprvse.exe with high cpu usage and note the PID. If it was the WMI service that had the high cpu, then you should already have it broken out to run in its own svchost process and note the PID of that svchost process. To confirm you have the right svchost process, you can run tasklist /svc from administrative command prompt and verify the PID noted in task manager and ensure it is the svchost process running winmgmt in it.
4. Run the following command: procdump –ma -s 60 -n 3 <PID>
Note: Replace <PID> with actual PID you documented for instance of wmiprvse.exe or for the svchost process running winmgmt exhibiting high memory usage
The above command will produce 3 dumps spaced 1 minute apart each in same directory you ran the procdump command from
5. Download the latest version of the Windows Sysinternals tool Process Explorer. http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx
6. If it was wmiprvse.exe that had the high CPU usage, then find the instance and right click on it and bring up the properties sheet. Click on the WMI Providers tab and document the listed providers
At this point you will now need to open a Support Incident Case with Microsoft to get the data analyzed to determine cause of high CPU usage.
Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.
Next up: WMI: How to Troubleshoot WMI High Handle Count
Scenario
Windows Management Instrumentation Service (Winmgmt) or WMI provider (wmiprvse.exe) is experiencing high handle count
Your first thing to do is check the Application Event log for following event:
Source: Microsoft-Windows-WMI
Event 5612 Wmiprvse.exe exceeding handle quota limit Event
WMI has stopped WMIPRVSE.EXE because a quota reached a warning value. Quota: %1 Value: %2 Maximum value: %3 WMIPRVSE PID: %4
If you find the above Event, you can try and bump up the handle quota limit to see if it resolves your issue. If it is a leak, then bumping limit will only mean it will take longer to reach the new limit. If it’s just load related, then bumping the limit could resolve the issue.
The event will tell you what the handle count was, and if it is higher than the 8192 value I suggest below. You can then skip this section and move on to data collection to figure out the cause of high handle count.
How to increase the handle quota limit for the WMI Provider Service
Note: you aren’t connecting to CimV2 or any other namespaces. It’s ROOT
Note: a double underscore __ precedes ProviderHostQuotaConfiguration
If after bumping up quota limit and wmiprvse still exceeding quota limit, accomplish following actions below. You will want to read through the rest in its entirety to ensure you get all of the necessary tools downloaded before taking any actions.
Configure System for Complete Memory Dump by referring to:
Windows 8 and Windows Server 2012 Automatic Memory Dump: http://blogs.technet.com/b/askcore/archive/2012/09/12/windows-8-and-windows-server-2012-automatic-memory-dump.aspx
Windows does not create a memory dump file when a Stop error occurs in Windows 8 or Windows Server 2012: http://support.microsoft.com/kb/2853466
Windows 2008, Windows Vista, Windows 7, Windows 2008 R2: http://support.microsoft.com/kb/969028
Windows Server 2003 and Windows XP: http://support.microsoft.com/kb/972110
Collect perfmon logging using logman method
Directions below will create 2 perfmon logs, one at a 5 minute interval (PerfLog-Long) and a short 5 second interval log (PerfLog-Short) and they will be placed in C:\Perflogs folder.
1. Click on Start
<<Start Search>>, enter "CMD.exe" w/o the quotation marks and then press Enter.
2. Copy and paste the following command into the command prompt window:
Logman.exe create counter PerfLog-Long -o "c:\perflogs\PerfLog-Long.blg" -f bincirc -v mmddhhmm -max 250 -c "\Cache\*" "\LogicalDisk(*)\*" "\Memory\*" "\Network Interface(*)\*" "\Paging File(*)\*" "\PhysicalDisk(*)\*" "\Processor(*)\*" "\Process(*)\*" "\Redirector\*" "\Server\*" "\Server Work Queues\*" "\System\*" -si 00:05:00
3. Start the log with:
Logman.exe start PerfLog-Long
4. Please stop the performance log as soon as the issue returns with the following command:
Logman.exe stop PerfLog-Long
1. Click on Start
<<Start Search>>, enter "CMD.exe" w/o the quotation marks and then press Enter.
2. Copy and paste the following command into the command prompt window:
Logman.exe create counter PerfLog-Short -o "c:\perflogs\PerfLog-Short" -f bincirc -v mmddhhmm -max 250 -c "\Cache\*" "\LogicalDisk(*)\*" "\Memory\*" "\Network Interface(*)\*" "\Paging File(*)\*" "\PhysicalDisk(*)\*" "\Processor(*)\*" "\Process(*)\*" "\Redirector\*" "\Server\*" "\System\*" "\Server Work Queues\*" "\Thread(*)\*" -si 00:00:05
3. Start the log with:
Logman.exe start PerfLog-Short
4. Please stop the performance log as soon as the issue returns with the following command:
Logman.exe stop PerfLog-Short
Please note that if you reboot the server, you will need to start the logs again as they will not automatically restart on boot.
Configure Handle Tracing
You probably just need the standalone version since we only need the debugging tool and not the whole WDK package.
If it is a svchost process showing high handle count, you can use Task Manager and add PID column, then identify which svchost process has the high memory usage. From there in a command prompt you can run tasklist /svc and look for the PID then identify if a single service is running in that svchost process or multiple services. If multiple services, it may become necessary to break each service out to run in its own svchost process to determine if it is the WMI service (winmgmt) that is causing the issue. From experience it will be the WMI service more times than not but not always, as such I would try to break it out first on its own and monitor to see if it is the one driving up high handle count in the shared svchost process.
WMI (Windows Management Instrumentation) service, you can break it out by accomplishing the following.
Break WMI Service out into its own unique svchost process
a. Open command prompt with elevated privileges
b. Run following command: sc config winmgmt type= own
c. Restart Wmi service
d. Run sc query winmgmt to ensure status of service now reflects “own” indicating running in its own svchost process
When issue had been resolved or no longer needing the service broken out into its own svchost process, place it back into the shared svchost process by running following command from command prompt:
sc config <service name> type= share
e. Restart the service or machine and verify result is Win32_SHARE_PROCESS when you run sc query winmgmt command again
f. Change command focus to system32 folder and run following command: copy svchost.exe wmisvchost.exe
g. From start run type in regedit and navigate to HKLM\System_CurrentControlSet\Services\Winmgmt
h. Modify existing ImagePath from %systemroot%\system32\svchost.exe -k netsvcs to %systemroot%\system32\wmisvchost.exe -k netsvcs
I. Restart wmi service with net stop winmgmt and net start winmgmt commands again
j. Verify you now see wmisvchost.exe process running by running tasklist or looking in task manager at process list
k. You would now substitute wmisvchost.exe in lieu of wmiprvse.exe in step 6. under Configure Handle Tracing above
Using debugger to attach to the process in windbg.exe and running !htrace –enable command
1. Launch WinDbg program from under Debugging Tools for Windows that you installed earlier.
2. Created folder c:\websymbols
3. Click on File-Symbol path and add the following symbol path to the debugger: SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols
4. Click on File-Save Workspace
Attach to process to accomplish handle tracking using htrace
To do this:
1. From Windbg - File - Attach a Process - Select the instance of wmiprvse.exe with high handle count
Note: If it is WMI Service (run tasklist /svc or Task Manager with PID column added first to find the PID of svchost.exe containing winmgmt which you should have broken out and uniquely named wmisvchost.exe per earlier directions)
2. Run following command from the debugger:
.logopen "C:\debug.log" then hit <ENTER> key
!htrace -enable 0x20000 then hit <ENTER> key
Note: By default, Windows Vista, Windows Server 2008, Windows 7 and Windows Server 2008 R2 keep a history of 4000 handles open and close operations.
With !htrace you can enable to keep a much higher history by doing the following:
!htrace -enable 0x20000
In this example, we are increasing the handle history to 131072 (decimal, 0x20000 hexadecimal)
!htrace –snapshot then hit <ENTER> key
g then hit <ENTER> key
3. Now, let the process run until the number of handle has increased a lot and gotten high.
Final htrace log
1. Break into debugger with Keyboard keys (Ctrl+Break)
2. Run following commands:
!htrace –diff then hit <ENTER> key
.logclose then hit <ENTER> key
.detach then hit <ENTER> key
3. Close WinDbg
Now complete the following actions once you have gotten your final htrace log
1. If high handle count is with wmiprvse, download the latest version of the Windows Sysinternals tool Process Explorer. http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx
2. Find the instance of wmiprvse.exe with high handle count and right click on it and bring up the properties sheet. Click on the WMI Providers tab and document the listed providers
3. If the WMI Service was the process with the high handle count, then dump out the WMI service process which should be wmisvchost.exe per previous directions and all instances of wmiprvse.exe using procdump. If it is wmiprvse.exe that is exhibiting the high handle count, then only need to dump out that instance and nothing else.
a. Download Windows Sysinternals tool called Procdump from URL: http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx
b. Open a command prompt with elevated or administrative rights and change to the directory were you saved Procdump
c. Open Task Manager and add the PID column view then go locate the instance of wmiprvse.exe with high memory usage and note the PID
d. Run the following command: procdump –ma <PID>
e. Note: Replace <PID> with actual PID you documented for instances of wmiprvse.exe and/or wmisvchost.exe as it applies based on directions above
4. Stop Perfmon logging
5. Do a complete memory dump of the machine
At this point with data in hand you will want to open a Support Incident with Microsoft to get the data analyzed to help determine cause of high handle count.
Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.
Good morning AskPerf! It’s been a while since our last post, and we apologize for that. We’ve been quite busy here on the Support side knocking out customer issues…
Any who, we have some upcoming blogs in the oven that need a little more time to bake. One of which is a short series on Windows Event Forwarding which I am very excited about. Look for that to come out in the coming months.
Even though we are commonly known as the Performance team, internally we are known as the Reliability team. Some of the technologies we support are as follows:
Windows Client/Server OS
There are many other smaller technologies, but these are the main ones.
Now back to our original topic: The Windows 10 Preview is available for download/testing. To get it, click the following link:
Finally, we always welcome feedback on topics you would like for us to blog about here on the AskPerf blog site.
-Blake
Hello AskPerf! Sanket here from the Windows Platforms team here to discuss an issue with Remote Desktop Services where RDP does not work when you try to connect from a remote machine. With that, let’s get started!
I’m sure most of you have come across the following message when connecting to a machine via RDP:
Remote Desktop Connection
This computer can't connect to the remote computer. Try connecting again. If the problem continues, contact the owner of the remote computer or your network administrator.
This is a generic that can be caused by numerous varying reasons. We have a fairly detailed troubleshooting KB article that talks about this error and what to do to fix it:
Assumptions are that most of you have followed this KB and resolved your issue. However, there could other reasons that could cause RDP to fail as well.
I recently worked an issue with same error where RDP from a remote machine was not connecting to a Windows 2012 Server. NOTE the same error can occur on previous OS versions as well.
There was a mystery as to what was changed on the server that could have caused this start. Possible assumptions were user intervention, or some application may have changed/removed certain permissions.
During the course of troubleshooting, we double-checked the KB article noted above, and noted the following Error events in the System Log:
Log Name: System
Source: Microsoft-Windows-TerminalServices-RemoteConnectionManager
Date: 7/27/2014 12:16:59 AM
Event ID: 1058
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: XXXXXXX
Description: The RD Session Host Server has failed to replace the expired self-signed certificate used for RD Session Host Server authentication on SSL connections.
The relevant status code was Access is denied.
This error indicates that there is already a Certificate in place, however there is no sufficient permissions, and/or the default permissions on “C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys” may have been modified.
&
Log Name: System
Source: Schannel
Date: --
Event ID: 36870
Task Category: None
Level: Error
Keywords:
User: SYSTEM
Computer: XXXXX
Description: A fatal error occurred when attempting to access the SSL server credential private key. The error code returned from the cryptographic module is 0x8009030D.
The internal error state is 10001.
There was a fatal error accessing the Private Key for secure communications.
At this point, I decided to capture a Process Monitor (Procmon) log on the destination server where the connection was going to. As you may already know, Procmon allows us to monitor/record real-time file system, Registry and process/thread activity on Windows Workstations/Servers.
Per the Procmon log, we found an “Access Denied” error to the following path:
“C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\f686aace6942fb7f7ceb231212eef4a4_”
The above cert key f686aace6942fb7f7ceb231212eef4a4_xxx is associated with RDS, and this GUID like number is the pair key for both the computer and user.
If you use the certutil -key command, you would see this Cert key with TSSecKeySet1:
f686aace6942fb7f7ceb231212eef4a4_xxxxxxxxxx: AT_KEYEXCHANGE
From the Procmon Logs:
12:39:53.5364585 AM lsass.exe 588 CreateFile C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\f686aace6942fb7f7ceb231212eef4a4_xxxx ACCESS DENIED Desired Access: Generic Read, Disposition: Open, Options: Sequential Access, Synchronous IO Non-Alert, Non-Directory File, Attributes: N/A, Share Mode: Read, Allocation Size: N/A,
Impersonating: NT AUTHORITY\NETWORK SERVICE
12:40:24.3692803 AM lsass.exe 588 CreateFile C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\f686aace6942fb7f7ceb231212eef4a4_xxxx ACCESS DENIED
Desired Access: Generic Read, Disposition: Open, Options: Sequential Access, Synchronous IO Non-Alert, Non-Directory File, Attributes: n/a, Share Mode: Read, Allocation Size: n/a, Impersonating: NT AUTHORITY\SYSTEM
12:40:23.9265708 AM svchost.exe 1012 CreateFile C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\f686aace6942fb7f7ceb231212eef4a4_xxxx ACCESS DENIED
Desired Access: Generic Read, Disposition: Open, Options: Sequential Access, Synchronous IO Non-Alert, Non-Directory File, Attributes: n/a, Share Mode: Read, Allocation Size: n/a
So, what are the default permissions? Well, you can use icacls to find this:
C:\>icacls C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\
Everyone :(R,W)
BUILTIN\Administrators :(F)
c:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\f686aace6942fb7f7ceb231212eef4a4_xxxxx
NT AUTHORITY\NETWORK SERVICE :(R)
NT AUTHORITY\SYSTEM :(F)
BUILTIN\Administrators ::(R)
In case if you want to grant permission using icals you can provide the same using following command :
icacls C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\f686aace6942fb7f7ceb231212eef4a4_xxxxx /grant " NT AUTHORITY\NETWORK SERVICE :( R)
icacls C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\f686aace6942fb7f7ceb231212eef4a4_xxxxx /grant " NT AUTHORITY\SYSTEM :(F)
icacls C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\f686aace6942fb7f7ceb231212eef4a4_xxxxx /grant " NT AUTHORITY\NETWORK SERVICE :(R )
Fig 1.1 (Permission in Windows Explorer)
As you can see above, the SYSTEM accounts needs the proper permissions. If these permissions have been changed, then they need put back to defaults. The certs under this key should be inheriting the above permissions from the parent folder MachineKeys.
You can restore permissions, grant the permissions back using icacls, or use the Windows Explorer GUI. Correcting the default permission on the cert should allow RDP to now work correctly.
Considering if this would have been easily reproducible, there is always an option to enable the Auditing on the cert key f686aace6942fb7f7ceb231212eef4a4_xxxxx under “C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys”. This can be done using the Security Tab on Properties of the cert key as seen in the screenshot below:
NOTE Adding Auditing on this object will log Events to the Security Event Logs. You will want to keep this enabled until you are able to reproduce the connection issue.
Hope you find this information helpful.
Additional Resources
Hello AskPerf! Pushing up a blog today to discuss the registry bloat issue that has been recently addressed in the following KB:
Computer cannot be restarted if the registry hives are larger than 2 GB
Symptom
Cause
This problem occurs because of the 2 GB size limit of the registry hives in x64-based version of Windows.
Resolution
Install this patch to resolve the issue.
When you get into this state, you may experience one of the following issues:
If this happens, KB2978366 should be installed.
With that, the following questions may come to mind:
Question: How does this issue occur?
Answer: There are many reasons that cause registry hives/keys to bloat. Some of the ones we have seen are related to KB2871131, which refers to the “..\Printers\DevModes2” key bloat. This hotfix does not “fix” the issue, but prevents it from occurring in the first place. You still have to clean the keys first. Additionally, there is a known issue with SQL Server 2012 SP1 that can cause the registry to hit the 2GB limit and put the machine in a no-boot state. Please see KB2793634 for more details on this.
Question: How do I prevent this issue in the first place?
Answer: There really is no good answer for this outside of installing the hotfixes noted above, and keeping a close eye out on your registry hives. You can use Performance Monitor however to monitor the “System\ % Registry Quota In Use” counter. If this counter gets over 50 %, then you should start investigating what registry keys/hives are growing.
% Registry Quota In Use is the percentage of the Total Registry Quota Allowed that is currently being used by the system. This counter displays the current percentage value only; it is not an average.
NOTE The following Registry hives point to their corresponding files:
Question: How do I fix this issue once the hotfix is installed?
Answer: After installing the hotfix, you may need to copy your Registry file to another machine that includes the hotfix. After you have cleared out the bloated entries (whitespace will remain), then simply load the hive up, and then unload it. This process will shrink your registry key back down pre-bloat. If a system is unbootable due to registry bloat install the hotfix on another system. Boot the problem system from DVD, copy the bloated registry hive to external storage, put on system with hotfix and use regedit to remove the bloated registry info and whitespace. The hive can then be copied back to problem system to allow it to boot normally.
Question: What happens if I see this problem on another OS version?
Answer: Simply copy your hive over to a Win 8/ Server 2012 machine that has this hotfix installed, then follow the steps above.
Question: Are there any tools I can use to troubleshoot this issue?
Answer: Coming Soon
Reference
How to Compress "Bloated" Registry Hives
-Blake
Hello folks,
One of our Engineering PMs that supports our Diagnostics and Automated solutions published a blog regarding Bing and how you can use it to answer your technical questions and provide automated solutions. Here is a brief overview:
Bing Technical Instant Answers provide concise answers to technical questions directly within search results and hopefully answer your question (or help you solve an issue) without you having to actually visit the web pages linked within the answer. The answers are triggered by specific search phrases, and they try to provide a unique benefit either by precisely matching your intent or by providing additional content related to your intent. In some cases, the instant answer will link to an automated fix or troubleshooter that you can run directly from the Bing search results. Microsoft will constantly be adding new technical answers, so if you have a technical problem with a Microsoft product or service try asking Bing to see if we have an instant answer for you!
Go check out his blog via the link below:
Using Bing for technical instant answers and automated solutions
-Blake
Hello Askperf! My name is Tingu, and today I’m going to talk about an interesting print migration issue I had a few weeks ago.
We had a case where an application server was running on Windows 2003, where more than 400 print queues were created. The port was created as a local port to forward the print job in case of a failure as noted in the “Transfer documents to another printer” Technet article.
The port was configured as \\printservername\printer. See example screenshot below:
Here, we were trying to move the application to a 2012 R2 server and wanted to migrate all the print queues to the new server. We used printbrmto migrate all of the local printers. But the problem we ran into is that it did not migrate the local ports.
When we started the migration, we did not see the local ports listed:
Additionally after the migration, the port was not present:
We tried to add the port manually, but gave us the error “port already exists”. Additionally, the registry shows that the printer is set to use the forwarder.
We really needed to get the local ports migrated as it can be a tedious task to re-create all the ports and map to their respective print queue.
We created a test lab and saw the same issue while migrating. It did not matter from/which OS we were migrating. During the migration, we saw an event ID 81 on the 2012 R2 server. (This event is not triggered if you are migrating to 2003 or 2008R2):
Log Name: Microsoft-Windows-PrintBRM/Admin
Source: Microsoft-Windows-PrintBRM
Date: 12/25/2014
Event ID: 81
Task Category: Restore
Level: Error
Keywords: Print Queue
User: Joe
Computer: 12345
Description:
Printbrm.exe (the Printer Migration Wizard or the command-line tool) failed to restore print queue test. The restore process will continue, skipping this queue. Error: 0x80070057 which is “invalid parameter”
Error: 0x80070057 which is pointing to “invalid parameter”
So what we determined is that when you use printbrm for migration, it will not migrate the local ports. The reason is that the local port is specific to the server, and it may cause conflicts or not work if you migrate it to a different server. But in our case it’s a forwarder, and we need it to be migrated.
Further testing revealed that if a local port to which the printer is mapped is already present on the destination server, then the migrated printers will use that local port for the printers.
For example: on the source server you have a printer mapped to LPT1, and the destination server has LPT1 port available; then after the migration, the printer will be set to use that port. We created a forwarder on the destination server for a test printer before migration, and after importing the printer, we see that the port is mapped accordingly.
Now the question is, how do we migrate multiple local ports at a time?
Here is what we did…
From the print management on a 2012 R2 server, we added the 2003 server. Then we exported the list of ports to a .csv file:
This gave us the list of all ports needing to be migrated. We then created a script to add the ports to the destination server. As in our case, the destination server was Windows 2012 R2 server, so we used the powershell command Add-PrinterPort.
We copied all of the required ports into notepad, and saved it as a .ps1 file:
We ran the .ps1 file as admin, and all of the ports got created on the destination server!
Note If you have already tried the migration before creating the ports on the destination server, it may give you the error ‘port already exists’ while running the powershell command. You may need to delete the printers migrated and restart the spooler and then retry the powershell command to complete the port creation.
After that, we followed the normal migration procedureand all printers got mapped to the correct port.
I hope this information will come in handy the next time you are working through a printer migration.
-Tingu
Hello Askperf! Anshuman here again with an interesting issue I worked a few weeks ago.
The following pop-up appeared on my workstation intermittently:
I then realized that I had the Send To OneNote printer set as my default printer.
The next time this occurred, I paused the print queue and noticed that the “Remote Desktop Redirected Printer Doc” document was getting spooled under my account. This was interesting because I had several remote desktop sessions opened to different machines from my workstation, and did not send any prints jobs from them.
So two questions came to mind:
1. Which RDS session is this coming from?
2. What was sending this print job?
I then thought to myself, “when in doubt, run Process Monitor!”
My first challenge was to figure out which server session this job was generated from. For this, I ensured that all the RDS sessions I established were using the command line option of of mstsc.exe (mstsc /v:servername). Next, I started process monitor on my workstation with a specific filter of “Process Name is mstsc.exe” and “Path contains .spl”. Since this issue was intermittent, I checked the “Drop filtered events” option. I also ensured that the Backing File option under File menu was pointing to a file, instead of Virtual Memory (pagefile). After a while the issue occurred, and procmon captured the following events:
One of the first things I noticed was the CloseFile operation immediately after the CreateFile operation. Typically, you will see a WriteFile operation in between these two operations. So mstsc is connecting to which server? That was easily found by examining the Command Line entry of mstsc captured in the pml file:
I logged into the problem server and launched procmon, ensuring that the Backing file option was set to point to a .pml file on a drive with enough space, and “Drop filtered events” was selected. Next I set up a filter “Path Contains tsclient” as well as “Path Contains RdpDr”. I then established an RDS session to the server from my work station and waited for the mysterious 0Kb print job. Once it happened, I had the following events in the pml file from the ProblemServer:
So there was an addon service that got installed on the printer server with a print driver. Disabling this ensured that those mysterious 0kb jobs ceased to occur.
-Anshuman
Hello AskPerf! Today’s post is a quick one that points to one of Bob' Golding's Windows Troubleshooting videos. He talks about how to download/run Dumpchk.exe on your dump files to check for corruption. Check it out below:
DumpChk – MSDN Link to more info
-Blake