.NET architecture, programming tips and tricks around Microsoft technology stack - Azure, WCF Services, SQL and strategy work.
Friday, February 23, 2007
Can MS SQL Server handle 1 million records per sec?
Load 1TB in less than 1 hour
OVERVIEW
This project was done using SQL Server 2005 Enterprise Edition (with SP1 beta) using the BULK INSERT command with 60 parallel input files, each of approximately 16.67 GB. The best time I got was 50 minutes. Hewlett Packard was kind enough to loan me some time one night on the machine they use to do the TPC-C tests. It is an Itanium2 Superdome with 64 1.6GHZ 9M (Madison) CPUs. For the insert tests the system was equipped with 256GB of RAM. The HP SAN configuration is rated at 14GB per second throughput. You can find the specific configurations at http://tpc.org/tpcc/results/tpcc_result_detail.asp?id=103082701. (Note: The configurations at this site may be updated periodically).
BEST PRACTICES and LESSIONS LEARNED
Here is a list of things we learned during this project that should be helpful in your projects.
Run as many load processes as you have available CPUs. If you have 32 CPUs, run 32 parallel loads. If you have 8 CPUs, run 8 parallel loads.
If you have control over the creation of your input files, make them of a size that is evenly divisible by the number of load threads you want to run in parallel. Also make sure all records belong to one partition if you want to use the switch partition strategy.
Use BULK insert instead of BCP if you are running the process on the SQL Server machine.
Use table partitioning to gain another 8-10%, but only if your input files are GUARANTEED to match your partitioning function, meaning that all records in one file must be in the same partition.
Use TABLOCK to avoid row at a time locking.
Use ROWS PER BATCH = 2500, or something near this if you are importing multiple streams into one table.
NOT FINISED YET! Check more on this blog to find out how to do test setups and other configuration steps that you need to take care of - http://blogs.msdn.com/sqlcat/archive/2006/05/19/602142.aspx
Cheers - D
Thursday, February 22, 2007
MSDN download free ... without MSDN subscription now!
Now, MSDN Library is free for download without any MSDN Subscription. It can be freely downloaded from: http://www.microsoft.com/downloads/details.aspx?FamilyID=373930cb-a3d7-4ea5-b421-dd6818dc7c41&DisplayLang=en
Happy Learning ! - Dj
MS CRM 3.0 - Coding and Testing tips...
Coding and Testing Tips -
This section contains development and testing tips that we use when working on Microsoft CRM projects. Hope you find them useful...
Microsoft .NET Framework version 2.0 versus version 1.1
Application Mode and Loader.aspx
Enabling the default Internet Explorer context menu
Accessing query string parameters
Referencing the Microsoft CRM assemblies or files
Authentication and coding with filtered views
Web.config file considerations
Server assembly impersonations
Authentication to test different users and roles
Enabling platform-level tracing
Enabling development errors
Microsoft .NET Framework Version 2.0
As documented in the SDK, Microsoft CRM 3.0 does support writing client-side code using Visual Studio .NET 2005 and the .NET Framework version 2.0 that accesses the CRMService and MetadataService Web services. However, Microsoft CRM 3.0 does not support callout or workflow assemblies written in Visual Studio .NET 2005 and the .NET Framework version 2.0. You should still continue to use Visual Studio .NET 2003 and the .NET Framework Version 1.1 when creating custom assemblies for callouts and workflow.
Please review the latest SDK for the most up-to-date information on the support for .NET Framework Version 2.0.
Application Mode and Loader.Aspx
Of course you've noticed that Microsoft CRM launches into a special Internet Explorer window that does not have the menu bar, address bar, toolbar, and so on. Microsoft CRM refers to this as the Application Mode, and it runs in this mode by default. Often, developers need these additional Internet Explorer features that application mode hides. You can launch Microsoft CRM in a standard Internet Explorer window by browsing to http://
Using the Loader.aspx page disables application mode for that single Web session. You can also permanently disable application mode for all users and all sessions by updating the Web.config file.
Disabling Application Mode
On the Microsoft CRM Web server, navigate to
Open the Web.config file in Notepad (or any text editor).
Look for the AppMode key, and change its value to Off.
Save the Web.config file.
Enabling the Default Internet Explorer Context Menu
In addition to running in Application Mode, Microsoft CRM modifies the standard Internet Explorer behavior by displaying its own context menu when you right-click in the application. Right-clicking a grid gives you options unique to Microsoft CRM, such as Open, Print, and Refresh List. However, right-clicking a form won't display a context menu like you would see on a normal Web page. When you're troubleshooting and debugging, you might find that you want to access the standard Internet Explorer context menu so that you can use features such as View Source, Properties, or Open In New Window. You can re-enable the Internet Explorer context menu by editing the Global.js file.
Re-Enabling the Internet Explorer Standard Context Menu
On the Microsoft CRM Web server, navigate to
Open the Global.js file in Notepad (or any text editor). Note: Do not double-click this file because it will attempt to execute the JavaScript file.
Right-click the file, and then click Edit.
Use your text editor's Find feature to locate the document.oncontextmenu() function.
Comment out the existing code in this function by adding /* and */ as shown in the following code. You can undo this change later by simply removing event.returnValue = true; line and the comment characters.
function document.oncontextmenu()
{
event.returnValue = true;
/*
var s = event.srcElement.tagName;
(!event.srcElement.disabled &&
(document.selection.createRange().text.length > 0
s == "TEXTAREA"
s == "INPUT" && event.srcElement.type == "text"));
*/
}
Save the file.
Open a page in Microsoft CRM and right-click it. You will see the familiar Internet Explorer context menu.
Caution Use this technique on development servers only. Do not modify the Global.js file in a production or staging environment; this unsupported change might cause unpredictable behavior. Microsoft CRM prevents use of the right-click context menu for the user's benefit and also to maintain a predictable navigation structure in the application interface.
Accessing Query String Parameters
When testing or debugging your code, you will frequently need the GUID of an entity record or want to see the query string parameters that Microsoft CRM passed to the window. You can access the URL of any page and view its GUID by pressing Ctrl+N on your keyboard. Internet Explorer opens a new window, and the address bar displays its query string parameters. You can also press the F11 key when viewing a record to toggle the display so that the URL address bar appears.
If you have re-enabled the Internet Explorer context menu, you can also get this information by right-clicking the page and selecting Properties. You can then copy the URL from the Properties dialog box.
Referencing the Microsoft CRM Assemblies or Files
You should never reference any Microsoft CRM assemblies other than Microsoft.Crm.Platform .Callout.Base.dll. You also should not import any of the JavaScript, style sheets, or behavior files into your project. Microsoft will not support this type of code reuse, and you will probably experience significant code problems when Microsoft CRM releases hotfixes, updates, or patches.

Caution Because you're not supposed to even reference any of the Microsoft CRM assemblies other than Microsoft.Crm.Platform.Callout.Base.dll, it should also be pretty obvious that you shouldn't attempt to modify any of the Microsoft CRM .dll files, either.
If you want to mimic any look or functionality of Microsoft CRM, you must re-create it yourself. The SDK provides a sample style sheet and a UI Style Guide, but if you want to review and understand the native script files, you can find most of them in <>\_common. If you want to review styles or code, copy these files to your own directory.
Authentication and Coding with Filtered Views
The SQL filtered views provide an excellent method for you to quickly and easily retrieve Microsoft CRM data by using standard SQL Server connections. However, because the filtered views rely on Integrated Windows authentication for data security, you must pay attention to how your code connects to the filtered views.
If you were to go under the hood of a filtered view in SQL and look at its script, you would see that it uses a custom function to enforce the security settings.
create function dbo.fn_FindUserGuid ()
returns uniqueidentifier
as
begin
declare @userguid uniqueidentifier
select @userguid = s.SystemUserId
from SystemUserBase s
where s.DomainName = SUSER_SNAME()
return @userguid
end
SUSER_SNAME() is a special function in SQL that returns the currently authenticated user, and the preceding function uses it to find the systemuserid identifier. If your code doesn't authenticate to SQL Server by using Windows authentication, your queries will never return any data, because all of the filtered views perform an inner join using systemuserid.
To connect to the database by using Windows authentication instead of SQL Server authentication, your connection string should include the following parameter: Integrated Security=SSPI. The full string would look something like this.
server=databaseserver;database=yourcustomdatabase;Integrated Security=SSPI
More Info Remember that you should never alter the Microsoft CRM databases by adding your own routines or stored procedures. Microsoft doesn't support this type of alteration, and it will probably result in the loss of your routines upon upgrades or hotfixes. Instead, you should create your own database to store your custom routines.
If you're calling your custom SQL routines from a Web page, you must configure additional settings to make sure that your Web page passes the integrated Windows credentials to SQL Server. First, the user should belong to the SQLAccessGroup, which will happen automatically when you add users to Microsoft CRM. Second, you must ensure that your Web application does not permit anonymous access by disabling anonymous access in IIS for your virtual directory or Web site. Finally, you must add or update the Web.config file in your custom web with the following nodes located in the
These keys ensure that IIS uses the Windows credential set when accessing a page in your virtual web.
As we explained earlier, another consequence of the integrated security features in filtered views is that you can't use filtered views in callouts or in workflow assemblies. The callout and workflow engine runs under the security context of the user, and, because you cannot impersonate to SQL Server from your connection string, you cannot retrieve data by using the filtered views. In these cases, we recommend that you use the QueryExpression class because the Web service does allow impersonation.
Web.Config File Considerations
If you create custom pages that work with Microsoft CRM, you should deploy them to a directory outside of the Microsoft CRM root and create a new Web site or a new virtual directory. The easiest way to leverage cross-site scripting (see Chapter 10 for more details) is to set up a virtual directory underneath the Microsoft CRM web. When you create a virtual directory, you will inherit the Web.config settings of the parent web, which in this case would be settings from the Microsoft CRM Web.config file.
If you add your custom web files to a virtual directory underneath the Microsoft CRM web, you might receive an error similar to this one when accessing your custom pages:
"File or assembly name Microsoft.Crm.Tools.ImportExportPublish, or one of its dependencies, was not found."
You have a couple of options for getting around this error. One is to register each of the assemblies that fail in the global assembly cache (GAC). The other is to remove these entries in your custom Web.config file. To remove the assemblies from your Web.config file, add the
Warning Make sure that you add this to your Web.config file. Do not alter these values in the Microsoft CRM Web.config file.
Server Assembly Impersonation
You might find that you need to use the credentials of the user making the request in your callouts or workflow assemblies. The SDK documents this numerous times, but, because it is such a useful technique, we want to mention here, too. This code snippet shows how to accomplish impersonation.
CrmService service = new CrmService();
service.Credentials = System.Net.CredentialCache.DefaultCredentials;
service.Url = "http://
// Get the current user ID.
WhoAmIRequest userRequest = new WhoAmIRequest();
WhoAmIResponse userResp = (WhoAmIResponse) service.Execute(userRequest);
service.CallerIdValue = new CallerId();
service.CallerIdValue.CallerGuid = userResp.UserId;
When you are using impersonation from a callout, the user context already contains the user ID for you. So, instead of making an additional request, you could simply use the following.
CrmService service = new CrmService();
service.Credentials = System.Net.CredentialCache.DefaultCredentials;
service.Url = "http://
service.CallerIdValue = new CallerId();
service.CallerIdValue.CallerGuid = userContext.UserId;
Enabling Development Errors in the Web.Config File
On the Microsoft CRM Web server, navigate to
Open the Web.config file in Notepad (or any text editor).
Look for the DevErrors key, and change its value to On.
Save the Web.config file.
Thanks to Mike Snyder and Jim Steger for writing down these tips in their book. You can refer to their book - Working with Microsoft Dynamics CRM 3.0 for more details. Thanks - D
MS CRM 3.0 - the world is talking...
Also, MS is coming up with TITAN as the latest release after ver 3.0. We will cover that as we mature :)
I have been lucky one to get a chance to work on MS CRM 3.0 a little. I have gurus around me who have vast experience on MS CRM, So probably i am going to blog more on this in coming days... as i learn more from all! Please feel free to add/edit or correct me if i go wrong anywhere. Thanks - Dipesh
Wednesday, February 21, 2007
SQL Service Broker in SQL 2005
What is service broker – It is a platform for async queued database applications... that's it :)
Steps involved to get started for SQL Service broker -
1 Create Message
2 Create Contract
3 Create Service (Sender/Receiver)
4 Create Queues
5 Assign Service to the Queue (to read queues)
6 Start a Dialog
7 Start a conversation (unique ID)
8 Send Messages
You can create Queues on same database on the same server or different servers.
You will need to create a Route specifying the Address to which the messages should be routed.
Important question though: Why messaging in database? May be the below points justify:-
1. Reliable service oriented arch
2. Parallel stored procs
3. Asynchronous triggers
4. Batch processing – Dialogs for ordering, Activation for parallelism, Distributed batch (multiple queues sitting remotely)
References –
Very good blog URL to understand SQL Service Broker - http://www.sqlservercentral.com/columnists/sSampath/anintroductiontotheservicebroker.asp
MSDN search – http://Msdn.microsoft.com/library/default.aspx – "Service broker" (tons of good information)
You can download SQL 2005 Express product to get started – http://msdn.microsoft.com/vstudio/express/sql/download/
lets get to the code to understand what we are talking about. You can follow these steps in SQL 2005--
--We will use adventure works as the sample database
USE AdventureWorks
GO
-- First, we need to create a message type. Note that our message type is
-- very simple and allowed any type of content
CREATE MESSAGE TYPE HelloMessage
VALIDATION = NONE
GO
-- Once the message type has been created, we need to create a contract
-- that specifies who can send what types of messages
CREATE CONTRACT HelloContract
(HelloMessage SENT BY INITIATOR)
GO
-- The communication is between two endpoints. Thus, we need two queues to
-- hold messages
CREATE QUEUE SenderQueue
CREATE QUEUE ReceiverQueue
GO
-- Create the required services and bind them to be above created queues
CREATE SERVICE Sender
ON QUEUE SenderQueue
CREATE SERVICE Receiver
ON QUEUE ReceiverQueue (HelloContract)
GO
-- At this point, we can begin the conversation between the two services by
-- sending messages
DECLARE @conversationHandle UNIQUEIDENTIFIER
DECLARE @message NVARCHAR(100)
BEGIN
BEGIN TRANSACTION;
BEGIN DIALOG @conversationHandle
FROM SERVICE Sender
TO SERVICE 'Receiver'
ON CONTRACT HelloContract
-- Send a message on the conversation
SET @message = N'Hello, World';
SEND ON CONVERSATION @conversationHandle
MESSAGE TYPE HelloMessage (@message)
COMMIT TRANSACTION
END
GO
-- Receive a message from the queue
RECEIVE CONVERT(NVARCHAR(max), message_body) AS message
FROM ReceiverQueue
-- Cleanup
DROP SERVICE Sender
DROP SERVICE Receiver
DROP QUEUE SenderQueue
DROP QUEUE ReceiverQueue
DROP CONTRACT HelloContract
DROP MESSAGE TYPE HelloMessage
GO
Happy learning :) - D
Thursday, February 15, 2007
Will Windows Vista support MSDE ?
Check this for SQL on Vista upgrade.. http://www.microsoft.com/sql/howtobuy/sqlonvista.mspx
Happy upgrading... - D
Wednesday, February 14, 2007
"How do i do" videos.... ASP.net AJAX
http://blogs.msdn.com/joestagner/archive/2007/02/13/free-asp-net-ajax-1-0-how-do-i-videos-updated-for-final-release.aspx
Cheers - D
Monday, February 12, 2007
http://www.microsoft.com/downloads/results.aspx?pocId=&freetext=keybinding&DisplayLang=en
Tuesday, February 06, 2007
ASP.NET AJAX 1.0 Released
ASP.NET AJAX 1.0 Released - ASP.NET AJAX is a free download with technical support provided by Microsoft. Create more sophisticated and interactive ASP.NET 2.0 Web sites that work across all browsers. Download today and start building a better Web!
The ASP.NET AJAX 1.0 Documentation can now be easily downloaded
ASP.NET AJAX Documentation
Microsoft ASP.NET AJAX documentation includes overviews, tutorials, samples, and API reference topics.
When you download the documentation, be sure to open the Readme.txt file for setup instructions. The download includes working samples to help you get the most out of ASP.NET AJAX.
Thansk - D
MSDN Webcast: Creating Dynamic Web Sites with ASP.NET 2.0 Web Parts (Level 300)
Thanks - D
Monday, February 05, 2007
Bill Gates launches Windows Vista, 2007 Microsoft Office...
Bill Gates launches Windows Vista, 2007 Microsoft Office
Watch the highlights as Bill Gates rolls out breakthrough versions of Microsoft's flagship products. You'll see how Windows Vista and the new Microsoft Office will transform the way people communicate, create and share content, and access information and entertainment.
Watch the webcast...
Cheers - D
LINQ - To be integrated with VS "Orcas"
Note that i made while reading was that you can use LINQ to write queries against varied multiple data sources like - SQL database, XML files or even the web service call.
You can check this implementation that supports LINQ calls to Amazon web service.
http://weblogs.asp.net/fmarguerie/archive/2006/06/26/Introducing-Linq-to-Amazon.aspx
Cheers - D
Thursday, December 21, 2006
XNA Game Studio Express 1.0 was officially released on December 11, 2006 (get it here).
You can download the XNA Game studio express 1.0 from http://msdn.microsoft.com/directx/XNA/default.aspx or you can check this independent site -
http://www.xnadevelopment.com/
This is free for download!!!
Explore the tutorials on XNA development www.xnadevelopment.com/ and learn how XNA uses .net Compact framework.
So, if you want some fun this X'mas go ahead and develop some games and impress your friends :)
Cheers- Dipesh
Friday, November 24, 2006
Gotcha - How to add uninstall option in the “User’s Program Menu” for web setup project?
The user doen’st have to go to control panel/add remove programs/ and then choose to uninstall the app, but just go to Start button, navigate to the app and choose uninstall!
Solution -
1. Create simple project UninstallApp.
2. Add in Main(){string[] arguments = Environment.GetCommandLineArgs();foreach(string argument in arguments){string[] parameters = argument.Split('=');if (parameters[0].ToLower() == "/u"){string productCode = parameters[1];string path = Environment.GetFolderPath(Environment.SpecialFolder.System);Process proc = new Process();proc.StartInfo.FileName = string.Concat(path,"\\msiexec.exe");proc.StartInfo.Arguments = string.Concat(" /i ", productCode);proc.Start();}}
3. Create new setup project
4. Add UninstallApp.exe to "Application Folder" in 'File System' part
5. In "User's Program menu" create shortcut to UninstallApp.exe and in properties of this shortcut in parameter 'arguments' insert value "/u=[ProductCode]".
6.Rebuild deployment project.
Thanks to Francisco Jileta and Michael Nielsen for pointing to us on this: http://gaaton.blogspot.com/2006/07/add-uninstall-and-him-shortcut-to-net.html
Hope it helps!
Thursday, October 19, 2006
Microsoft Visual Studio Code Name "Orcas" Community Technology Preview - Development Tools for .NET Framework 3.0
Keep exploring... :)
"IIS 7.0"/ IE 7.0 release much awaited...
** Check this excellent talk on "Channel 9" with Scott Guthrie here: Scott Guthrie - Talking ASP.NET and IIS 7.0 http://channel9.msdn.com/ShowPost.aspx?PostID=44084
*** Check out the IIS 7.0 show here: The .NET Show: IIS7.0 (http://msdn.microsoft.com/theshow/episode.aspx?xml=theshow/en/Episode056/manifest.xml )
* IE 7.0 can be explored and downloaded here: www.microsoft.com/windows/ie/downloads/
Thanks - D
Live webcasts free from "Redmond"...
Sunday, September 17, 2006
2 simple steps to enable notifications from SQL 2005 to ASP.net 2.0
A> Change the SQLDependency from your DB:table name to CommandNotification
B> And in the Global.asax file under Application_Start event start the SQL Dependancy. That's it!
System.Data.SQLClient.SQLDependancy.Start(ConnectionString)
1> NO POLLING FROM THE ASP.net runtime
2> AVOIDS DATABASE TRIPS UNNECESSARILY
3> SQL SERVER 2005 automatically sends notification whenever there is change in the data (at table level)
4> SILENT communication between SQL engine and .net runtime
5> Performance benefit compared to ASP.net polling
Note: You need to have SQLCacheDependancy enabled = True in web.config file
Cheers - DJ
Wednesday, August 30, 2006
Thursday, August 24, 2006
Automatically Update Visual Basic 6 Applications with .NET and ClickOnce
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnvs05/html/atupvb6ap2.asp
Monday, July 17, 2006
Keep Sites Running Smoothly By Avoiding These 10 Common ASP.NET Pitfalls
Keep Sites Running Smoothly By Avoiding These 10 Common ASP.NET Pitfalls -
- LoadControl and Output Caching
- Sessions and Output Caching
- Forms Authentication Ticket Lifetime
- View State: The Silent Perf Killer
- SQL Server Session State: Another Perf Killer
- Uncached Roles
- Profile Property Serialization
- Thread Pool Saturation
- Impersonation and ACL Authorization
- Don’t Just Trust It—Profile Your Database!
The (un)official guide to SQL Reporting Services 2000 & 2005 interop
Check this blog for questions surrounding differences between SQL 2000 and SQL 2005 reporting services: http://blogs.msdn.com/bimusings/archive/2005/12/13/503110.aspx
Thursday, June 15, 2006
Back after a break...
I recently attended 2 day Microsoft Tech Ed session in Mumbai, India... overall good one. Spoke lot about their new products and dozen more which could not make sense to me.,. i am lil dumb may be :)
Anyways what interested me was WinFX track which i attended last day... the presenter spoke lot on WPF, WWF and WCF.
List of new products outlined:
Windows Vista
Windows WorkFlow - not sure if it will be called as .net Framework 3.0
BizTalk 2006
Sharepoint 2007
C# 3.0 / VB 9.0 (not sure when it is going to be released, but plans)
New version of Visual Studio guess code named Orcass
LINQ
Ofcourse along with VS 2005, SQL 2005 and VSTS coming off the market !!!
btw, Did you check the Internet Explorer 7.0 beta copy?
Check this ... http://www.microsoft.com/windows/ie/default.mspx
Cheers - D
Monday, April 24, 2006
ahhhhhhhhhh! long time, i have been missing... my fault...but i will try to be much over this henceforth.
hmmm, so check this Microsoft Certified Architect Program that identifies top industry experts in IT Architecture. These architects can employ multiple technologies to solve business problems and provide business metrics and measurements to describe the success or failure of the projects they drive. The program is currently in beta, and is expected to open to the architect community in early 2006.
http://www.microsoft.com/learning/mcp/architect/default.asp
Also, "Explore the latest posts from architecture bloggers as well...."
http://msdn.microsoft.com/architecture/shareideas/architectureblog/default.aspx
Cheers - D