Batch Processing

/**
l'm trying to improve my batch processing routines by
creating a generic method which l can apply on most of
my jobs.Reason Being l'm working with large Data Sets
This is what l'm trying to achieve
1.Get The Rowcount of the Source
2.Split It Into manageable batches
3.Loop through the batches Until the last one is Done
**/
How do l intergrate these two so it can automatically pick up the
batchsizes and process until the last batch is done.
--Outer Loop
Declare @.RowCount Int
Declare @.LoopIteration Int
Declare @.BatchSize Int
--Get The RowCount Of the Table
Select @.RowCount = (Select Count(*) From tbTest)
--
Select @.LoopIteration = @.RowCount/2000000
Select @.BatchSize = @.RowCount/@.LoopIteration
--Select Distinct Client,Convert(Varchar(15),'') As CltNo
-- ,Convert(Int,0) As Status Into tbtest
from tbAddress
--Inner Loop --Works when run Manually
SET ROWCOUNT @.BatchSize--100000 --54 Secs
WHILE 1 = 1
BEGIN
Begin Transaction
Update tbTest
Set CltNo = Client + 1
,Status = 1 Where Status = 0
Commit
if @.@.rowcount = 0
break
END
SET ROWCOUNT 0
--Select * from tbTest Where Status = 1
Message posted via http://www.webservertalk.comWhy do you care what the table size is? You should pick a batch size that
can be accomplished in a short enough time so that it does not interfere
with other users and can have minimal overhead. It usually doesn't matter
if the table is 10M rows or 100K rows. Your batch size should be the same.
What that size is can only be found with proper testing and adjustment.
Usually 10K is a good place to start as it is small enough to finish quickly
and large enough to nibble at the work load. IF you check the @.@.ROWCOUNT
and break when it is 0 then you don't have to worry about how many loops
there will be. Another comment is that there is no need to wrap the Update
in a Begin - Commit. Each update is ATOMIC all by itself. If you update
10K rows in a single update statement all 10K will either work or they wont.
There will not be any middle ground. By adding it in there you now have to
put error checking and handle possible open trans.
Andrew J. Kelly SQL MVP
"Raymond M via webservertalk.com" <forum@.nospam.webservertalk.com> wrote in
message news:a911f5b29ab44d0b837051f840966d55@.SQ
webservertalk.com...
> /**
> l'm trying to improve my batch processing routines by
> creating a generic method which l can apply on most of
> my jobs.Reason Being l'm working with large Data Sets
> This is what l'm trying to achieve
> 1.Get The Rowcount of the Source
> 2.Split It Into manageable batches
> 3.Loop through the batches Until the last one is Done
> **/
> How do l intergrate these two so it can automatically pick up the
> batchsizes and process until the last batch is done.
> --Outer Loop
> Declare @.RowCount Int
> Declare @.LoopIteration Int
> Declare @.BatchSize Int
>
> --Get The RowCount Of the Table
> Select @.RowCount = (Select Count(*) From tbTest)
> --
> Select @.LoopIteration = @.RowCount/2000000
> Select @.BatchSize = @.RowCount/@.LoopIteration
>
>
> --Select Distinct Client,Convert(Varchar(15),'') As CltNo
> -- ,Convert(Int,0) As Status Into tbtest
> from tbAddress
> --Inner Loop --Works when run Manually
> SET ROWCOUNT @.BatchSize--100000 --54 Secs
> WHILE 1 = 1
> BEGIN
> Begin Transaction
> Update tbTest
> Set CltNo = Client + 1
> ,Status = 1 Where Status = 0
> Commit
> if @.@.rowcount = 0
> break
> END
> SET ROWCOUNT 0
> --Select * from tbTest Where Status = 1
> --
> Message posted via http://www.webservertalk.com