Monitoring SharePoint Usage through an ASP.NET Web Application

Introduction

It is a common requirement to gather information around the Windows SharePoint site usage. Organizations prefer to find out the popular sites in their organizations are or what external WSS sites attract the most number of visitors. This will provide information necessary for an organization to identify the important sites in their environment and properly manage them with extra resources if necessary.

One of the common complains around accessing theses data are the inflexibility of gathering information in a central location. The administrator will have to navigate to the selected SharePoint site administration pages to access the data. This will become a more time consuming exercise when the administrator need to collect usage data from larger number of SharePoint sites.

In this article, I am looking at creating an ASP.NET web application that will populate a collection of SharePoint sites in a drop down list for a given SharePoint site collection. The web application will display the usage details of the selected SharePoint site from the list. This will help the SharePoint administrator to gather all SharePoint usage data from a central location with out have to navigate many different SharePoint site locations.

Usage Analysis Data Web Report

Figure 1 display a Usage Analysis data for a Windows SharePoint site. The users have the option of selecting a monthly or a daily report.

Figure 1: Monthly Usage Analysis Report

Usage Analysis Processing Reports

First of all let's have a look at what is Usage Analysis Reports in SharePoint. Theses reports will help the organizations to determine how the web sites are used in their environment. The data is taken from IIS Logs on the front end web servers and stored in to temporary files. The data is merged in to the content database on the back end SQL Servers when the daily log processing takes place.

The Usage Analysis is not enabled by default when the organization deploys the Windows SharePoint Services Sites. Organizations should enable the Usage Analysis logging process on the servers when they require gathering usage information. The logs files are created daily and a flag is attached to mark that it has been processed.

These logs files are not automatically deleted, they are preserved in "local_drive (C) :\Windows\system32\LogFiles\W3SVC*" where * indicates the Internet Information Server (IIS) instance number of the virtual server as displayed in Figure2. Organizations should consider the advantages against the disk space storage before enabling the Usage Analysis service in their environment. The organization can stop logging process' any time they require to do so.

Figure 2: Preserved log files folder structure with W3SVC* format

By default the log files are located at "local_drive (C) :\Windows\system32\LogFiles\STS" directory. Separate folders are created under above directory for each virtual server and separate folders for each day as displayed in the Figure 3.

Figure 3: Separate directories for each virtual server and for each day

Organizations can configure the above Log file store path for their own preferred path and create up to 30 log files. (Please look at Setting up Usage Analysis process for more details.) If an organization decides to store the log files in their preferred location, they should grant Read, Write and Update rights permissions for STS_WPG user group for the specified folder. Without the permissions, the usage log files cannot be created or updated by IIS.

Setting up Usage Analysis Processing

Administrators can control the setting of Usage Analysis process using the SharePoint Central Administration page. The user must be an administrator on the local server or a member of the SharePoint Administrators group to configure the analysis processing. If the organization adds a new virtual server after the analysis service been configured, they will need to reconfigure the analysis service to collect the data on the newly added virtual server.

• Click "Start -> All Programs -> Administrative Tools -> SharePoint Central Administration"
  
• This will open the "Windows SharePoint Services Central Administration" page
  
• Select the "Configure usage analysis processing" from the "Component Configuration" section as displayed in figure 4
  
  Figure 4: Component Configuration section on the central administration page
  
• The Configure Usage Analysis Processing page will open as displayed in Figure 5
  
  Figure 5: Configure Usage Analysis Processing page
  
• Select "Enable logging" check box in "Logging Settings" section
  
• The administrator can select a preferred file locations for the log files in "Log file location" box or use the default location C:\WINDOWS\system32\logfiles\STS
  
• Enter the number of log files to create in the "Number of log files to create" box. This can be between 1 and 30.
  
• Select "Enable usage analysis processing" check box in "Processing Settings" section.
  
• Select the time range that analysis process should run from "Run processing between these times daily". By default this is set to start at 1am daily. The administrator can allocate a time period that won't affect the demand on the site.
  
• Click "OK" to finish the configuration.
  

View Usage Analysis Reports

The user must be an administrator or have the View Usage Data right for a site to view the site usage reports. The reports are available through Site Administration page.

The usage data is collected and stored for the entire site collection on a server at a time. The users can see the total number of hits for a site collection on the Site Collection Usage Summary page and for more detailed information, Site Usage Report page for individual sites or sub sites usage information.

View Site Usage Report

• Navigate to the site you prefer to view the report.
  
• Click Site Settings on top tool bar.
  
• Click Go to Site Administration link under Administration section as displayed in Figure 6
  
  Figure 6: Site Settings page
  
• Click View site usage data link under Management and Statistic section in Top-Level Site Administration page as displayed in Figure 7
  
  Figure 7: Top-Level Site Administration page
  
• The Site Usage Report page will open as displayed in Figure 8
  
  Figure 8: Site Usage Report page
  

Site usage reports are useful for identifying which content on Windows SharePoint Services sites are being heavily used or used very little. This will help organizations to understand which sites are candidates for archiving and which sites should be kept online. In addition, this report contains information regarding how much storage space WSS sites are using. This page provides a report that contains following information:

• Pages that have been accessed on that site, including document libraries
  
• Users that have access the site
  
• Operating system of the user accessing the site
  
• Browser type
  
• Referrer URL
  
• Data can be displayed by monthly or daily summary format
  

Figure 9: Different reports available

The users can select a report option and a daily or monthly option to generate a report. Figure 9 displays a monthly report of all the pages accessed and different kind of reports options available.

Code Example

The web page contains a text box to enter the SharePoint site collection URL. The appropriate sub site will be listed in a dropdown list when user clicks the Submit button. The user then have the option of viewing the daily or monthly usage report of a selected site.

First of all you will need to add the Microsoft.SharePoint.dll to your web application reference list. This will give us the access to the SharePoint Object Model.

Then instantiate the SPSite object as displayed below. The absolute URL is passed in through the txtWSSSiteUrl text box. This will populate site collection for the given URL.

//Get the site collection
SPSite mySiteCollection = new SPSite(txtWSSSiteUrl.Text);

Then to access an individual site, instantiate the SPWeb object as displayed below. I am passing in the site name as a parameter.

//Get the details of the selected WSS site
SPWeb site = mySiteCollection.AllWebs[siteName];

After constructing the site SPWeb object, developers can access the information of the site usage data using the public method "GetUsageData" of the SPWeb object as displayed in code example.

GetUsageData(Microsoft.SharePoint.Administration.SPUsageReportType, Microsoft.SharePoint.Administration.SPUsagePeriodType) Method.

The GetUsageData method of the SPWeb class returns a data table that contains information about the usage of a Web site based on the specified type of report and time interval.

SPUsageReportType Enumeration

The SPUsageReportType enumeration specifies the type of information returned in a usage report for a SharePoint site.

The following table shows the members of the SPUsageReportType enumeration and a brief description

Name	Description
browser	The type of Web browser used to visit the SharePoint site. All usage data refers specifically to visits from referring URLs external to the site.
os	The operating system used on the client computer. All usage data refers specifically to visits from referring URLs external to the site.
refUr	External URLs through which users navigated to the SharePoint site.
url	URLs of pages that are visited or of pages for lists that are updated. Discussions about a page are counted as hits on that page.
user	Users who visited the site.

SPUsagePeriodType Enumeration

The SPUsagePeriodType enumeration specifies the time interval on which a usage report for a Web site is based.

The following table shows the members of the SPUsagePeriodType enumeration and a brief description

Name	Description
day	Returns usage information for each day during the past 31 days starting from the previous day
lastMonth	Summarizes usage information for the last 31 days relative to the previous day

I am binding the data table return from GetUsageData property to a DataGrid control to display the information.

Accessing User daily report

//Users who visited the site
DGUsers.DataSource = site.GetUsageData(SPUsageReportType.user, SPUsagePeriodType.day);

Accessing User monthly report

//Users who visited the site
DGUsers.DataSource = site.GetUsageData(SPUsageReportType.user, SPUsagePeriodType.lastMonth);

Accessing Browser daily report

//The type of browsers used to visit the site
DGBrowser.DataSource = site.GetUsageData(SPUsageReportType.browser, SPUsagePeriodType.day);

Accessing Browser monthly report

//The type of browsers used to visit the site
DGBrowser.DataSource = site.GetUsageData(SPUsageReportType.browser, SPUsagePeriodType.lastMonth);

Accessing Operating System daily report

//The Operating System used in client computer
DGOs.DataSource = site.GetUsageData(SPUsageReportType.os, SPUsagePeriodType.day);

Accessing Operating System monthly report

//The Operating System used in client computer
DGOs.DataSource = site.GetUsageData(SPUsageReportType.os, SPUsagePeriodType.lastMonth);

Accessing refUrl daily report

//External URL client used to navigate to SharePoint site
DGRefUrl.DataSource = site.GetUsageData(SPUsageReportType.refUrl, SPUsagePeriodType.day);

Accessing refUrl monthly report

//External URL client used to navigate to SharePoint site
DGRefUrl.DataSource = site.GetUsageData(SPUsageReportType.refUrl, SPUsagePeriodType.lastMonth);

Accessing url daily report

//URL's of pages visited
DGUrls.DataSource = site.GetUsageData(SPUsageReportType.url, SPUsagePeriodType.day);

Accessing url monthly report

//URL's of pages visited
DGUrls.DataSource = site.GetUsageData(SPUsageReportType.url, SPUsagePeriodType.lastMonth);

Deploying the Web Application to the SharePoint Portal Server

• Create the "UsageAnalysisData" Virtual directory under the portal web site in IIS
  
• Map the virtual directory to the physical file path (this should be done as apart of above step)
  
• Open the SharePoint Central Administration pages by clicking "Start > Administration tools > SharePoint Central Administration"
  
• Click "Windows SharePoint Services" link on the left hand side
  
• Select "Configure Virtual server setting" link from "Virtual Server Configuration" group
  
• Select your Portal server from the list
  
• Click "Defined managed path" link under Virtual Server Management group in Virtual Server setting page
  
• Type name of the virtual directory you created in first step in the Path section of Add new Path section
  
• Select "Exclude path" radio button under the type section
  
• Click "Ok"
  
• Then navigate to http://portal_site_name/UsageAnalysisData/MonitoringPage.aspx
  

Figure 10: A list of sub sites of the site collection

Figure 11: Daily Usage Analysis Report of a Windows SharePoint Site

Conclusion

SharePoint Administrators should be able to use this article as a starting point and develop their SharePoint Usage Analysis data gatherer web application according to their requirements.

Update SharePoint UserInfo List with More Active Directory Info

• Download source code - 439 KB
  

Introduction

A little while back, I downloaded a Web Part that was supposed to show the current user's weather. It was coded to query the SiteUserInfoList and pull back the user's WorkCity, WorkState, and WorkCountry properties so that it could correctly locate the user. I installed the Web Part and found that it didn't work. Those properties are in Moss' User Profile but not in the Site collection's SiteUserInfoList. The person who coded the Web Part obviously didn't like to do testing. So, what was I to do? I didn't want to resort to Moss' User Profile because I want all my Web Parts to work for non-Moss implementations. So, I decided to just write a job that would run every night to update the SiteUserInfoList with more Active Directory properties.

Also, are you tired of getting "User does not exist or is not unique" errors when adding users to groups? Well, this code will also update the title of every user that is missing in Active Diretory to "XX - <previousTitle>". This is an easy way to expose to you that the user is missing in Active Directory so that when you are using the User Selector dialog, you can prevent yourself from getting that error.

Installation

First, copy the build directory to the desktop of each server in your farm. Run "Install AD Updator.bat" as Administrator. Type in the name of your Web Application you want to install to, click Enter. Make sure you only run this on one server at a time.

The Code

The code runs through all the users in SharePoint and then pulls their information from Active Directory. It then takes the info from Active Directory and updates specific fields like Title, Phone, and etc. Then, it stores the rest of the info (WorkCity, WorkZip, and etc.) in the property bag of the user's UserInfo item.

Collapse Copy Code

using System;

using System.Collections.Generic;

using System.Text;

using Microsoft.SharePoint.Administration;

using Microsoft.SharePoint;

using System.Diagnostics;

using Microsoft.SharePoint.Utilities;

using System.DirectoryServices;

using System.DirectoryServices.ActiveDirectory;

using System.Security.Principal;

namespace Mullivan.SharePoint.Jobs

{

public class AdUserInfoUpdateJobDefinition : SPJobDefinition

{

internal const string _JOBNAME = "AD User Info Update Job";

public AdUserInfoUpdateJobDefinition () : base() { }

public AdUserInfoUpdateJobDefinition(SPWebApplication webApp)

: base(_JOBNAME, webApp, null, SPJobLockType.Job)

{

this.Title = _JOBNAME;

}

public override void Execute(Guid targetInstanceId)

{

try

{

DirectoryEntry domain = GetDomainEntry();

if (domain == null)

throw new Exception("Domain not found.");

foreach (SPSite site in this.WebApplication.Sites)

{

using (SPWeb web = site.RootWeb)

{

SPListItemCollection userItems = web.SiteUserInfoList.Items;

for (int i = 0; i < userItems.Count; i++)

{

try

{

double progress = ((double)(i + 1)) /

(double)userItems.Count;

UpdateProgress(Convert.ToInt32(progress * 100));

SPListItem userItem = userItems[i];

 

SPUser user = web.SiteUsers.GetByID(userItem.ID);

if (user == null)

throw new Exception(string.Format(

"User account {0} not found in site {1}.",

userItem.Name, site.Url));

DateTime dtUserItemUpdated =

(DateTime)userItem["Modified"];

if (IsPerson(userItem) && !IsSystem(user))

{

AdUserInfo userInfo = GetUserInfo(user, domain);

if (userInfo == null || !userInfo.IsActive)

{

string jobTitle = (string)userItem["JobTitle"];

if (string.IsNullOrEmpty(jobTitle))

jobTitle = string.Empty;

if (!jobTitle.StartsWith("XX - "))

{

jobTitle = string.Format("XX - {0}", jobTitle);

userItem["JobTitle"] = jobTitle;

userItem.Update();

}

}

else

{

object updateFlag =

userItem.Properties["AdUpdateFlag"];

if (userInfo.LastModified > dtUserItemUpdated

|| updateFlag == null)

{

userItem.Properties["AdUpdateFlag"] = 1;

if (userInfo.Email != null)

{

userItem["EMail"] = userInfo.Email;

user.Email = userInfo.Email;

}

if (userInfo.Department != null)

userItem["Department"] = userInfo.Department;

if (userInfo.JobTitle != null)

userItem["JobTitle"] = userInfo.JobTitle;

else

{

string val = (string)userItem["JobTitle"];

if (val != null)

{

if (val.StartsWith("XX - "))

userItem["JobTitle"] =

val.Substring(5, val.Length - 5);

}

}

if (userInfo.FirstName != null)

userItem["FirstName"] = userInfo.FirstName;

if (userInfo.LastName != null)

userItem["LastName"] = userInfo.LastName;

if (userInfo.WorkPhone != null)

userItem["WorkPhone"] = userInfo.WorkPhone;

if (userInfo.Office != null)

userItem["Office"] = userInfo.Office;

if (userInfo.WorkZip != null)

userItem.Properties["WorkZip"] =

userInfo.WorkZip;

if (userInfo.WorkCity != null)

userItem.Properties["WorkCity"] =

userInfo.WorkCity;

if (userInfo.WorkState != null)

userItem.Properties["WorkState"] =

userInfo.WorkState;

if (userInfo.WorkCountry != null)

userItem.Properties["WorkCountry"] =

userInfo.WorkCountry;

userItem.Update();

user.Update();

}

}

}

}

catch (Exception ex)

{

Logging.ServiceLog.LogException(_JOBNAME, ex);

}

}

web.Dispose();

}

site.Dispose();

}

}

catch (Exception ex)

{

Logging.ServiceLog.LogException(_JOBNAME, ex);

}

}

private bool IsSystem(SPUser user)

{

if (user.ID.Equals(1073741823))

return true;

if (user.LoginName == null)

return true;

if(user.LoginName.ToLower().StartsWith("nt authority"))

return true;

if (user.LoginName.ToLower().StartsWith("system"))

return true;

return false;

}

private AdUserInfo GetUserInfo(SPUser user, DirectoryEntry domain)

{

string id = user.Sid;

bool localMachine = domain.Path.StartsWith("WinNT");

string userFlagProperty = "userAccountControl";

if (localMachine)

userFlagProperty = "UserFlags";

DirectoryEntry deUser = FindUser(id, domain);

if (deUser != null)

{

AdUserInfo adUserInfo = new AdUserInfo();

adUserInfo.IsActive = true;

if (localMachine)

{

//For testing purposes... Production Environment should be using

// Active Directory

adUserInfo.LastModified = DateTime.Now;

string value = GetValue("FullName", deUser);

if (!string.IsNullOrEmpty(value))

{

string[] vals = value.Split(new char[1] { ' ' }

, StringSplitOptions.RemoveEmptyEntries);

if (vals.Length > 0)

adUserInfo.FirstName = vals[0];

if (vals.Length > 1)

adUserInfo.LastName = vals[vals.Length - 1];

}

adUserInfo.WorkCity = "St Louis";

adUserInfo.WorkState = "MO";

adUserInfo.WorkZip = "63141";

 

}

else

{

DateTime dtModified = DateTime.Now;

if (DateTime.TryParse(GetValue("whenChanged", deUser),

out dtModified))

adUserInfo.LastModified = dtModified;

else

adUserInfo.LastModified = DateTime.Now;

adUserInfo.LastName = GetValue("sn", deUser);

adUserInfo.FirstName = GetValue("givenName", deUser);

adUserInfo.Name = GetValue("sAMAccountName", deUser);

adUserInfo.Office = GetValue("physicalDeliveryOfficeName", deUser);

adUserInfo.WorkPhone = GetValue("telephoneNumber", deUser);

adUserInfo.Department = GetValue("department", deUser);

adUserInfo.Email = GetValue("mail", deUser);

adUserInfo.JobTitle = GetValue("title", deUser);

adUserInfo.WorkCity = GetValue("l", deUser);

adUserInfo.WorkState = GetValue("st", deUser);

adUserInfo.WorkCountry = GetValue("c", deUser);

adUserInfo.WorkZip = GetValue("postalCode", deUser);

}

string userAC = GetValue(userFlagProperty, deUser);

int userValue = 0;

if(int.TryParse(userAC, out userValue))

{

try

{

AdUserAccountControl userAccountControl =

(AdUserAccountControl)userValue;

adUserInfo.IsActive =

//Make sure it's not disabled

((userAccountControl & AdUserAccountControl.ACCOUNTDISABLE)

!= AdUserAccountControl.ACCOUNTDISABLE)

//Make sure it's a normal account

&& ((userAccountControl &

AdUserAccountControl.NORMAL_ACCOUNT) ==

AdUserAccountControl.NORMAL_ACCOUNT);

}

catch (Exception ex)

{

Logging.ServiceLog.LogException(_JOBNAME, ex);

}

}

return adUserInfo;

}

else

return null;

}

private string GetValue(string propertyName, DirectoryEntry deUser)

{

if (deUser.Properties.Contains(propertyName))

{

PropertyValueCollection pvc = deUser.Properties[propertyName];

if (pvc.Count > 0)

{

object objValue = pvc[0];

if (objValue != null)

return objValue.ToString();

}

}

return null;

}

private DirectoryEntry FindUser(string id, DirectoryEntry domain)

{

if (!domain.Path.StartsWith("WinNT"))

{

DirectorySearcher search = new DirectorySearcher(domain);

search.Filter =

string.Format("(&(objectClass=person)(objectSid={0}))", id);

SearchResult result = search.FindOne();

if(result != null)

return result.GetDirectoryEntry();

}

else

{

foreach (DirectoryEntry de in domain.Children)

{

SecurityIdentifier si = new SecurityIdentifier(

(byte[])de.Properties["objectSid"][0], 0);

if (string.Compare(si.Value, id, true) == 0)

return de;

}

}

return null;

}

private DirectoryEntry GetDomainEntry()

{

try

{

return Domain.GetComputerDomain().GetDirectoryEntry();

}

catch(Exception ex)

{

Logging.ServiceLog.LogException(_JOBNAME, ex);

#if SULLYSERVER

DirectoryEntry localMachine = new DirectoryEntry(

string.Format("WinNT://{0},Computer", Environment.MachineName));

return localMachine;

#else

return null;

#endif

}

}

private bool IsPerson(SPListItem userItem)

{

string contentType = userItem.ContentType.Name;

if (!contentType.Equals("Person"))

return false;

return true;

}

private void Trace(string message)

{

System.Diagnostics.Trace.WriteLine(message, _JOBNAME);

}

}

}

We can now pull this information out and use it in our Web Parts by doing the following (this code is an example taken from the WeatherWebPart that I modified):

Collapse Copy Code

private bool SetUserLocation()

{

try

{

SPWeb web = SPContext.Current.Web;

SPUser u = web.SiteUsers[web.CurrentUser.LoginName];

SPList userList = web.SiteUserInfoList;

SPListItem uItem = userList.Items.GetItemById(u.ID);

 

if (uItem != null)

{

string strZip = uItem.Properties["WorkZip"] as string;

if (!string.IsNullOrEmpty(strZip))

this.Zip = strZip;

return true;

}

}

catch

{

this.Zip = DEFAULT_ZIP;

}

return false;

}

Create an hyper link to Info path form

<SCRIPT>

var L_NewDocumentRuntimeError_Text = L_NewFormLibTb1_Text;

var L_NewDocumentError_Text = L_NewFormLibTb2_Text;

function WPQ3createNewDocument()

{

var strTemplate = "http://hysps01:2009/test/New%20Purchase%20Request%20Form/Forms/template.xsn";

var strSaveLocation = "http://hysps01:2009/test/New%20Purchase%20Request%20Form";

var strProgID = "SharePoint.OpenXMLDocuments";

createNewDocumentWithProgID(escapeProperlyCore(strTemplate, true),

makeAbsUrl(strSaveLocation), strProgID, true);

}

</SCRIPT>

<a href="javascript:WPQ3createNewDocument();">Click here to add a new contact</a>