Showing posts with label High Availability. Show all posts
Showing posts with label High Availability. Show all posts

Wednesday, May 12, 2010

My first SQL 2008 cluster on vSphere

Recently I had to install a SQL 2008 cluster on Windows 2008 cluster which was virtualized. I learned some valuable lessons I want to share with you.

First of all there is this new feature in the VM Tools called shared folders. Make sure it is off. It causes an error message and the description has nothing to do with the cause.

The second thing is if you want to install service pack 1 for SQL Server, slipstream it. There is a bug that crashes your installation and you can't actually remove it. The term slipstream is a not really the correct term but it works.

First you unpack the service pack with the /x option and then you need to run /x64/setup/1033/sqlsupport.msi and run it. The next step is to start the SQL Server setup and start it from the command line with the parameter /PCUSource=

More info on slipstream can be fount at http://support.microsoft.com/kb/955392.

Wednesday, December 3, 2008

Lessons learned ... clustering SQL 2005 on a Windows 2008 (x64) cluster

Last couple of days have been verry stressful. I've just finished a clustered install of SQL2005 on a Win2008. I'll be talking in this post about the issues i had.

The first problem I had was getting to know Win2008. I know according to Microsoft that everything has improved but the challenge for us professionals is to keep up with the fact that some things don't have the same name or are completely on a different location. So I lost quite some valuable time on calling around asking a Windows system engineer with the questions "where can I find ..." and "How do I do ...". Yes, I felt like a complete idiot.

First I had to set up the MSDTC cluster resource, in Win2008 there is a wizard for that. I used it and it worked fine.

Then I had to add my clustered disks ... formatting took like eternity. No I 'm kidding but it took a while and was very impatient to go on.

Once all that was done I organised the cluster resources and started the SQL Server install. It went as planned. I ticked the box to tell it was a cluster install and continued on my quest.

The first error message I got was this one







It is actually the full-text search service that is down and it is simply solved by installing SQL 2005 SP2. After clicking ok I ran into a next error.












It is stupid but I needed to install Visual Studio 2005 SP1 (I only found x86) to solve this one. I think this happens because the SQL Server Managament Studio is written in 32-bit and I was working on x64. I know in Windows 2003 this is no issue but appearently on Windows 2008 it becomes one.

After fixing this I installed SQL 2005 SP2 and that has a problem on its own. I noticed that I was not able to make maintenance plans. I got these nice screenshots:


























This is what happend: for some reason, don't ask me why it seems that the resource database had trouble to update in SP2. SP2 appearently changed some things ...

If you run in to this, you can verify the version of your resource databases with this query:

SELECT SERVERPROPERTY('ResourceVersion');

When I ran that query It answered me 9.00.1399 which is the RTM version (RTM is how it is shipped the first day your version is sold). Okay, that was usefull info but how to solve this was a mystery to me. I crused around on the information high way and found out that all it took wat to manually run the queries that are located in the "sysdbupg" script. This script can be found in "C:\Program Files\Microsoft SQL Server\MSSQL\Install". I ran it and got the maintenance plans back.

The last issue i had was an issue with database mail. Luckely for me there are nice people who blog and at Jean-Pierre Paradis' Blog I found what I was looking for. I got an activation failure.

I just copied Jean-Pierre's solution and it worked. I repeat it here if for some reason you would not be able to get to his blog.

First create the text file DataBaseMail90.exe.config in the \MSSQL\Binn folder of your SQL Instance (ex: \Program Files\Microsoft SQL Server\MSSQL.1\MSSQLSERVER\MSSQL\Binn). with the following content :










Next thing to do is to load this in SSMS:

USE msdb;
GO
INSERT INTO [msdb].[dbo].[sysmail_configuration]
(
[paramname]
,[paramvalue]
,[description]
)
VALUES
(
N'ReadFromConfigurationFile'
,N'1'
,N'Send mail from mail server in configuration file'
);

Then I replaced this stored procedure:

USE [msdb]
GO
/****** Object: StoredProcedure [dbo].[sp_sysmail_activate] Script Date: 12/01/2008 15:41:40 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER OFF
GO
-- sp_sysmail_activate : Starts the DatabaseMail process if it isn't already running
--
ALTER PROCEDURE [dbo].[sp_sysmail_activate]
AS
BEGIN
DECLARE @mailDbName sysname
DECLARE @mailDbId INT
DECLARE @mailEngineLifeMin INT
DECLARE @loggingLevel nvarchar(256)
DECLARE @loggingLevelInt int
DECLARE @parameter_value nvarchar(256)
DECLARE @localmessage nvarchar(max)
DECLARE @rc INT

EXEC @rc = msdb.dbo.sysmail_help_configure_value_sp @parameter_name = N'DatabaseMailExeMinimumLifeTime',
@parameter_value = @parameter_value OUTPUT
IF(@rc <> 0)
RETURN (1)

--ConvertToInt will return the default if @parameter_value is null or config value can't be converted
--Setting max exe lifetime is 1 week (604800 secs). Can't see a reason for it to ever run longer that this
SET @mailEngineLifeMin = dbo.ConvertToInt(@parameter_value, 604800, 600)

--Try and get the optional logging level for the DatabaseMail process
EXEC msdb.dbo.sysmail_help_configure_value_sp @parameter_name = N'LoggingLevel',
@parameter_value = @loggingLevel OUTPUT

--Convert logging level into string value for passing into XP
SET @loggingLevelInt = dbo.ConvertToInt(@loggingLevel, 3, 2)
IF @loggingLevelInt = 1
SET @loggingLevel = 'Normal'
ELSE IF @loggingLevelInt = 3
SET @loggingLevel = 'Verbose'
ELSE -- default
SET @loggingLevel = 'Extended'

SET @mailDbName = DB_NAME()
SET @mailDbId = DB_ID()

EXEC @rc = master..xp_sysmail_activate @mailDbId, @mailDbName, @mailEngineLifeMin, @loggingLevel
IF(@rc <> 0)
BEGIN
SET @localmessage = FORMATMESSAGE(14637)
exec msdb.dbo.sysmail_logmailevent_sp @event_type=3, @description=@localmessage
END
ELSE
BEGIN
SET @localmessage = FORMATMESSAGE(14638)
exec msdb.dbo.sysmail_logmailevent_sp @event_type=0, @description=@localmessage
END

RETURN @rc
END

with this one

USE [msdb]
GO
/****** Object: StoredProcedure [dbo].[sp_sysmail_activate] Script Date: 08/13/2008 11:59:49 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
-- sp_sysmail_activate : Starts the DatabaseMail process if it isn't already running
--
ALTER PROCEDURE [dbo].[sp_sysmail_activate]

AS
BEGIN
DECLARE @mailDbName sysname
DECLARE @mailDbId INT
DECLARE @mailEngineLifeMin INT
DECLARE @loggingLevel nvarchar(256)
DECLARE @loggingLevelInt int
DECLARE @parameter_value nvarchar(256)
DECLARE @localmessage nvarchar(max)
DECLARE @readFromConfigFile INT
DECLARE @rc INT

SET NOCOUNT ON
EXEC sp_executesql @statement = N'RECEIVE TOP(0) * FROM msdb.dbo.ExternalMailQueue'

EXEC @rc = msdb.dbo.sysmail_help_configure_value_sp @parameter_name = N'DatabaseMailExeMinimumLifeTime',
@parameter_value = @parameter_value OUTPUT
IF(@rc <> 0)
RETURN (1)

--ConvertToInt will return the default if @parameter_value is null or config value can't be converted
--Setting max exe lifetime is 1 week (604800 secs). Can't see a reason for it to ever run longer that this
SET @mailEngineLifeMin = dbo.ConvertToInt(@parameter_value, 604800, 600)

EXEC msdb.dbo.sysmail_help_configure_value_sp @parameter_name = N'ReadFromConfigurationFile',
@parameter_value = @parameter_value OUTPUT
--Try to read the optional read from configuration file:
SET @readFromConfigFile = dbo.ConvertToInt(@parameter_value, 1, 0)

--Try and get the optional logging level for the DatabaseMail process
EXEC msdb.dbo.sysmail_help_configure_value_sp @parameter_name = N'LoggingLevel',
@parameter_value = @loggingLevel OUTPUT

--Convert logging level into string value for passing into XP
SET @loggingLevelInt = dbo.ConvertToInt(@loggingLevel, 3, 2)
IF @loggingLevelInt = 1
SET @loggingLevel = 'Normal'
ELSE IF @loggingLevelInt = 3
SET @loggingLevel = 'Verbose'
ELSE -- default
SET @loggingLevel = 'Extended'

SET @mailDbName = DB_NAME()
SET @mailDbId = DB_ID()

EXEC @rc = master..xp_sysmail_activate @mailDbId, @mailDbName, @readFromConfigFile,
@mailEngineLifeMin, @loggingLevel
IF(@rc <> 0)
BEGIN
SET @localmessage = FORMATMESSAGE(14637)
exec msdb.dbo.sysmail_logmailevent_sp @event_type=3, @description=@localmessage
END
ELSE
BEGIN
SET @localmessage = FORMATMESSAGE(14638)
exec msdb.dbo.sysmail_logmailevent_sp @event_type=0, @description=@localmessage
END

RETURN @rc
END


Start the procedure with the follwing SQL command :

EXEC msdb.dbo.sysmail_start_sp;

Finally that was the last issue I had. Thanks to everybody who helped me, like always in our business it is a matter of looking at a problem with as many as possible.

Saturday, November 29, 2008

Database mirroring on SQL 2005 SP2

Howdy folks,

This week I went together with a colleague to a customer where there were some troubles with the database mirroring on SQL 2005. It a high available mirror and the problem was it didn't work 100% of the time. Some times something like a split brain occurs. The database is recovering as well on the mirror as on the principal.

The thing we noticed was in the mirroring monitor that the server who had the principal role didn't have a successful connection to the witness server.

First thing we checked was the network. The situation goes as follows. The servers are in workgroup mode and there is a dedicated Gigabit connection between those server to sent the transactions to both servers. All servers were also defined in the host file so even when the DNS goes down it should work.

After crusing down the Internet I found a post in a forum of somebody with the same problem. The problem was solved for this guy by rebooting the system. So we restarted the SQL service of the witness server and it worked.

So that needed some further investigation. We created a new database, made it mirror and the same scenario ... no witness on the principal, only for that database. We restarted the SQL service on the witness and it worked :-)

Then we did the ultimate test, we stopped the endpoint on the server for which the server was mirror for the databases in production and principal for our test database. (So there was no impact for production). First time we tested it, everything went fine and the test database failed over (and we wrote a record in it). The second time we tried it, it failed and a split brain like situation occured. Okay, there was no problem to bring the test database online and since it is in sync mode no transaction could have been written to one side and not to the other.

The odd thing is that the production server had to be turned off that evening and the automatic fail over worked without any problems.

So some further investigations will be required. Currently we are thinking in the direction of cummulative updates. When we find it I'll make a post about it.

Thursday, August 7, 2008

A night at an ISP

Recently I've spent the night at one of Belgium's bigger Internet service providers. The ISP had had some trouble with their databases last December and I had to implement database mirroring.

In the beginning of July I had created a test database for their IT people so they could play with it. And now, the time had come to implement it for all their databases as a test to adapt their programming and make it fail-over aware.

There were some specifics as the mirror had to be synchronous and encrypted and it had to be the same port on each server.

So here are my findings:
  • Use 2005 SP2, it figures but I prefer to mention it ;)
  • You need the database to be in full recovery mode
  • Watch out for the auto close option, it runs the fun
  • You need a full backup and a transaction log backup
It actually went pretty smooth since everything (mirror endpoints, encryption and witness) was already there from the previous month.

The mirror wizard doesn't use the full qualified network name for the principal server so at the end it proposes to start mirroring but it fails because you have to manually adapt the principal server.

The only thing that was a real problem was 1 database. For some reason it failed time after time and the error message was that it was unable to connect to the witness or mirror server.

The cause was one app that writes constantly in the database and since it took about 15 minutes to move the backup and restore it with no recovery on the witness it was not possible to create the mirror.

To work around this I made the full backup, restored it with no recovery and then I made the transactional backup. I had the permission to take the database offline once I made the transaction log backup had finished and restored it on the mirror . Once I had put the database back online the mirroring was no problem at all.

We ran some tests and everything went fine. The only thing my customer still has to do is create maintenance plans on the mirror (for some weird reason you can't mirror those) and alter his apps.

At the break of rush hour we all went home for some sleep :).

An update: 10 days later and something went wrong, for some reason one database went suspect on the principal.