Hi guys
I have a small problem. I want to use BCP with in a transaction it's this
first I execute a SQL Statement against the database and delete some data,
then
I run the BCP utility.
my problem is how to use these two with in a single transaction!
please help me
I appreciate your help and time!
thanks,
-ArunaI haven't tested it myself, but see if BULK INSERT is usable in a transactio
n.
Tibor Karaszi, SQL Server MVP
http://www.karaszi.com/sqlserver/default.asp
http://www.solidqualitylearning.com/
"Aruna Tennakoon" <aruna.tennakoon@.gmail.com> wrote in message
news:%23L2k7OdWFHA.3840@.tk2msftngp13.phx.gbl...
> Hi guys
> I have a small problem. I want to use BCP with in a transaction it's this
> first I execute a SQL Statement against the database and delete some data,
> then
> I run the BCP utility.
> my problem is how to use these two with in a single transaction!
> please help me
> I appreciate your help and time!
> thanks,
> -Aruna
>|||Hi Tibor,
Thanks for your reply. I tried the BULK INSERT statment but, it didn't work
because it need the text file in the SQL Server.I am using client PC to do
this.
Please send me if you have any sample.
Thanks again
-Aruna
"Tibor Karaszi" <tibor_please.no.email_karaszi@.hotmail.nomail.com> wrote in
message news:OY36rDfWFHA.3760@.TK2MSFTNGP15.phx.gbl...
>I haven't tested it myself, but see if BULK INSERT is usable in a
>transaction.
> --
> Tibor Karaszi, SQL Server MVP
> http://www.karaszi.com/sqlserver/default.asp
> http://www.solidqualitylearning.com/
>
> "Aruna Tennakoon" <aruna.tennakoon@.gmail.com> wrote in message
> news:%23L2k7OdWFHA.3840@.tk2msftngp13.phx.gbl...
>
2012年3月27日星期二
2012年3月25日星期日
BCP Transaction Does not roll back
Hi All,
I'm trying to import some data using the BCP command line utility.
I've set the maxerrors switch to 0. In case of an exception such as a
cast exception the import fails leaving the table state dirtied.
ie...with partial data imported.
From what I understand from a few other posts the transaction logs
only store the space alocated and not the actual data.
How does one ensure that the import is done in a transaction ?
Any suggestions/ideas will be great.
Regards,
Avinash
Can you post the command you are running
HTH
Jasper Smith (SQL Server MVP)
http://www.sqldbatips.com
I support PASS - the definitive, global
community for SQL Server professionals -
http://www.sqlpass.org
"Avinash" <avinashraj@.gmail.com> wrote in message
news:f22f61b0.0412050250.3d6ef381@.posting.google.c om...
> Hi All,
> I'm trying to import some data using the BCP command line utility.
> I've set the maxerrors switch to 0. In case of an exception such as a
> cast exception the import fails leaving the table state dirtied.
> ie...with partial data imported.
> From what I understand from a few other posts the transaction logs
> only store the space alocated and not the actual data.
> How does one ensure that the import is done in a transaction ?
> Any suggestions/ideas will be great.
> Regards,
> Avinash
I'm trying to import some data using the BCP command line utility.
I've set the maxerrors switch to 0. In case of an exception such as a
cast exception the import fails leaving the table state dirtied.
ie...with partial data imported.
From what I understand from a few other posts the transaction logs
only store the space alocated and not the actual data.
How does one ensure that the import is done in a transaction ?
Any suggestions/ideas will be great.
Regards,
Avinash
Can you post the command you are running
HTH
Jasper Smith (SQL Server MVP)
http://www.sqldbatips.com
I support PASS - the definitive, global
community for SQL Server professionals -
http://www.sqlpass.org
"Avinash" <avinashraj@.gmail.com> wrote in message
news:f22f61b0.0412050250.3d6ef381@.posting.google.c om...
> Hi All,
> I'm trying to import some data using the BCP command line utility.
> I've set the maxerrors switch to 0. In case of an exception such as a
> cast exception the import fails leaving the table state dirtied.
> ie...with partial data imported.
> From what I understand from a few other posts the transaction logs
> only store the space alocated and not the actual data.
> How does one ensure that the import is done in a transaction ?
> Any suggestions/ideas will be great.
> Regards,
> Avinash
2012年3月19日星期一
bcp operation within transaction
hello,
I have a problem where I am calling the BCP utility to write a table to a file. I then need to delete the rows of the table. but not all of them. This all works fine. I've been asked to place this into a transaction..incase a piece fails. When I do that...SQL server hangs. I must shutdown SQL Server.
Any idea why that would happen. I am using the xp_cmdshell stored procedure to invoke bcp utility within a stored procdure. The procedure is executed every 15 minutes to provide files to an outboard system.
Salik.BCP operation is not affected by a transaction ...
This might help...
http://www.sqlteam.com/forums/topic.asp?TOPIC_ID=31640|||Well...
wouldn't you say that it's an implicit TRANS?
I mean if the bcp fails in the middle (in it's own spid) then that rolls back..
But as Enigma has said...if you do
BEGIN TRAN
master..xp_cmdshell 'bcp...
ROLLBACK TRAN
IF the bcp was successful, then you won't be able to roll it back...
EDIT:
And...
When I do that...SQL server hangs
I would say that's not true...do sp_lock
I would say your spids are blocking each other...
been there...done that....|||Well ... its a thing to be left alone ...
as you say ... been there .. done that (discussion)|||My main issue is that I am writing out data based on a 'events' table which tracks the changes users make to a product table. from the time I have built the strings and inserted them into a temporary table and then output the file, then removed the 'events' - a user could change a field in a product I am writing out - which would be missed by the system. Not good.
The only thing I can think of is to lock the appropriate rows in the product table for the duration. WHich I'm not aware of the syntax.
Any other strategies?
Salik.|||Data is fluid ny bature...what's the difference if it's updated during your export, or immediatley after...the data will still be changed...
And is your issue a matter of concurrency? Why build a temp table?
Also why not schedule a batch window? Grab the date from the system...use that as the window close date, grab all rows added or updated between the last time the window close and this windoes close...
So if they update it during your export it wouldn't be part of this batch...
How long is your transaction...and how many rows on average do you export?|||The output data is '*' delimited for another system and requires padding of numbers, calculation etc.. that's why I build the strings into a temporary table and then ship the temp table out using bcp.
We use triggers to track the events a user makes to a product. on certain fields a product event is created. and on others a price event. The price events are held util evening processing as they can only be sent out once a day. the product files are output every 15 minutes. There are four different files output at the same time, ecah containing slightly different data.
The events table prevents duplicates ie: we don't append. if a existing price event exists we don't add another row.
when I have finished the bcp the appropriate rows are removed from the events table.
sooo - between the time the row for the product has been read and the time I delete the events row for that product. and event could happen...no duplicate would be added..and the new data not written out and the system thinks it has.
I could snatch the rows from the events table into a temptable. delete them and if an error occurs add them back in to the events table...
I have a problem where I am calling the BCP utility to write a table to a file. I then need to delete the rows of the table. but not all of them. This all works fine. I've been asked to place this into a transaction..incase a piece fails. When I do that...SQL server hangs. I must shutdown SQL Server.
Any idea why that would happen. I am using the xp_cmdshell stored procedure to invoke bcp utility within a stored procdure. The procedure is executed every 15 minutes to provide files to an outboard system.
Salik.BCP operation is not affected by a transaction ...
This might help...
http://www.sqlteam.com/forums/topic.asp?TOPIC_ID=31640|||Well...
wouldn't you say that it's an implicit TRANS?
I mean if the bcp fails in the middle (in it's own spid) then that rolls back..
But as Enigma has said...if you do
BEGIN TRAN
master..xp_cmdshell 'bcp...
ROLLBACK TRAN
IF the bcp was successful, then you won't be able to roll it back...
EDIT:
And...
When I do that...SQL server hangs
I would say that's not true...do sp_lock
I would say your spids are blocking each other...
been there...done that....|||Well ... its a thing to be left alone ...
as you say ... been there .. done that (discussion)|||My main issue is that I am writing out data based on a 'events' table which tracks the changes users make to a product table. from the time I have built the strings and inserted them into a temporary table and then output the file, then removed the 'events' - a user could change a field in a product I am writing out - which would be missed by the system. Not good.
The only thing I can think of is to lock the appropriate rows in the product table for the duration. WHich I'm not aware of the syntax.
Any other strategies?
Salik.|||Data is fluid ny bature...what's the difference if it's updated during your export, or immediatley after...the data will still be changed...
And is your issue a matter of concurrency? Why build a temp table?
Also why not schedule a batch window? Grab the date from the system...use that as the window close date, grab all rows added or updated between the last time the window close and this windoes close...
So if they update it during your export it wouldn't be part of this batch...
How long is your transaction...and how many rows on average do you export?|||The output data is '*' delimited for another system and requires padding of numbers, calculation etc.. that's why I build the strings into a temporary table and then ship the temp table out using bcp.
We use triggers to track the events a user makes to a product. on certain fields a product event is created. and on others a price event. The price events are held util evening processing as they can only be sent out once a day. the product files are output every 15 minutes. There are four different files output at the same time, ecah containing slightly different data.
The events table prevents duplicates ie: we don't append. if a existing price event exists we don't add another row.
when I have finished the bcp the appropriate rows are removed from the events table.
sooo - between the time the row for the product has been read and the time I delete the events row for that product. and event could happen...no duplicate would be added..and the new data not written out and the system thinks it has.
I could snatch the rows from the events table into a temptable. delete them and if an error occurs add them back in to the events table...
2012年2月23日星期四
bcp and transaction problem
Hi,
We are creating a procedure in which we use bcp
(xp_cmdshell) to archive out the data for particular
month. When I execute procedure without begin tran/commit
tran it works fine and bcp out the data from table to hard
disk. But when i try to use transaction handling using
begin tran i see it to be waiting and dbcc inputbuffer of
waiting process shows:
SET FMTONLY ON select * from db..bcptable SET FMTONLY OFF
Why sql server is starting seperate process within
procedure ...IS there any special way to handle
transaction when using bcp and xp_cmdshell in the
procedure?
This is the only procedure running on machine.
Thanks
--harvinderharvinder,
> We are creating a procedure in which we use bcp
> (xp_cmdshell) to archive out the data for particular
> month. When I execute procedure without begin tran/commit
> tran it works fine and bcp out the data from table to hard
> disk. But when i try to use transaction handling using
> begin tran i see it to be waiting and dbcc inputbuffer of
> waiting process shows:
> SET FMTONLY ON select * from db..bcptable SET FMTONLY OFF
> Why sql server is starting seperate process within
> procedure ...IS there any special way to handle
> transaction when using bcp and xp_cmdshell in the
> procedure?
Bcp is running as a separate connection, not as a part of your
transaction. This is normal. It is being blocked by the connection
that opened the transaction. You cannot wrap a call to bcp in a
transaction.
Linda
We are creating a procedure in which we use bcp
(xp_cmdshell) to archive out the data for particular
month. When I execute procedure without begin tran/commit
tran it works fine and bcp out the data from table to hard
disk. But when i try to use transaction handling using
begin tran i see it to be waiting and dbcc inputbuffer of
waiting process shows:
SET FMTONLY ON select * from db..bcptable SET FMTONLY OFF
Why sql server is starting seperate process within
procedure ...IS there any special way to handle
transaction when using bcp and xp_cmdshell in the
procedure?
This is the only procedure running on machine.
Thanks
--harvinderharvinder,
> We are creating a procedure in which we use bcp
> (xp_cmdshell) to archive out the data for particular
> month. When I execute procedure without begin tran/commit
> tran it works fine and bcp out the data from table to hard
> disk. But when i try to use transaction handling using
> begin tran i see it to be waiting and dbcc inputbuffer of
> waiting process shows:
> SET FMTONLY ON select * from db..bcptable SET FMTONLY OFF
> Why sql server is starting seperate process within
> procedure ...IS there any special way to handle
> transaction when using bcp and xp_cmdshell in the
> procedure?
Bcp is running as a separate connection, not as a part of your
transaction. This is normal. It is being blocked by the connection
that opened the transaction. You cannot wrap a call to bcp in a
transaction.
Linda
标签:
archive,
bcp,
creating,
database,
execute,
microsoft,
mysql,
oracle,
particular,
procedure,
server,
sql,
transaction,
xp_cmdshell
2012年2月18日星期六
bcp - Transaction LogFile Size issue
Hi Experts,
I am having new issue again,
My task is to copy data from one table to another table residing in
different database.
Table happenes to be extreamly large. (Contains around 15 million
rows.)
I tried several ways (SQL query, SSIS packages,etc...)
I found BCP utility suits my requirement. So planned for BCP.
I am trying following ways (Two steps)
1. bcp <MyFirstDB.TableName> out <MyFlatFilePath> -n -T
2.bcp <MySecondDB.TableName> in <MyFlatFilePath> -n -T
(Basically, Copying data from source to flat file and from there to
DestinationTable.)
Here problem is, My transaction Log file (MySecondDB_log.ldf) grows
like hell on second command. It grows upto 8 GB (max. free space That
I have on disk
).
My datatransfer will be incomplete because of no space on HardDisk.
PLease let me know, where I am going wrong,if you have better method,
how can i optimize my data transfer. (My log file grows by 2 MB, not
with %ge)
Thank you in advance,
Sriharsha Karagodu.
Are you changing the second database's recovery to bulk?
On Mar 17, 9:56Xam, sriharsha.karag...@.gmail.com wrote:
> Hi Experts,
> I am having new issue again,
> My task is to copy data from one table to another table residing in
> different database.
> Table happenes to be extreamly large. (Contains around 15 million
> rows.)
> I tried several ways (SQL query, SSIS packages,etc...)
> I found BCP utility suits my requirement. So planned for BCP.
> I am trying following ways (Two steps)
> 1. bcp X<MyFirstDB.TableName> out <MyFlatFilePath> -n -T
> 2.bcp X<MySecondDB.TableName> in <MyFlatFilePath> -n -T
> (Basically, Copying data from source to flat file and from there to
> DestinationTable.)
> Here problem is, My transaction Log file (MySecondDB_log.ldf) grows
> like hell on second command. It grows upto 8 GB (max. free space That
> I have on disk
).
> My datatransfer will be incomplete because of no space on HardDisk.
> PLease let me know, where I am going wrong,if you have better method,
> how can i optimize my data transfer. (My log file grows by 2 MB, not
> with %ge)
> Thank you in advance,
> Sriharsha Karagodu.
|||On Mar 17, 6:59Xpm, Sean <ColdFusion...@.gmail.com> wrote:
> Are you changing the second database's recovery to bulk?
> On Mar 17, 9:56Xam, sriharsha.karag...@.gmail.com wrote:
>
>
>
>
>
>
> - Show quoted text -
Actually I read about Changing the Recovery Property. But Where Do I
Get that option?
When I Do Property of Database--> options-->recovery, This will have
three options like,
TonrnPageDetection,
CheckSum,
None.
So, Not sure where will get option to change the recovery to BULK.
please guide me.
|||You can do it visually, but the syntax goes like this:
ALTER DATABASE [database name]
SET RECOVERY [either: FULL | BULK_LOGGED | SIMPLE]
Also, to see what the current recovery model is run: SP_HELPDB
[database name]
1. Run SP_HELPDB [database name], and note the model used.
2. ALTER DATABASE [database name] SET RECOVERY BULK_LOGGED
3. Run SP_HELPDB [database name] to double check the settings
4. Run bcp import
5. ALTER DATABASE [databasename] SET RECOVERY (output of step 2)
6. SP_HELPDB [database name] to double check
|||Shriharsha,
When BCPing in so much data, it is also good to use the batch size operator
to control the transaction size. Such as:
bcp ... -b 50000
This will break up your bcp into about 300 batches, which will speed it up
and gives you more transaction log control. You could then (if necessary)
run extra BACKUP LOGs during the bcp in.
RLF
"Sean" <ColdFusion244@.gmail.com> wrote in message
news:ba587a67-ef18-452e-88a2-d6132234fc5c@.t54g2000hsg.googlegroups.com...
> You can do it visually, but the syntax goes like this:
> ALTER DATABASE [database name]
> SET RECOVERY [either: FULL | BULK_LOGGED | SIMPLE]
> Also, to see what the current recovery model is run: SP_HELPDB
> [database name]
> 1. Run SP_HELPDB [database name], and note the model used.
> 2. ALTER DATABASE [database name] SET RECOVERY BULK_LOGGED
> 3. Run SP_HELPDB [database name] to double check the settings
> 4. Run bcp import
> 5. ALTER DATABASE [databasename] SET RECOVERY (output of step 2)
> 6. SP_HELPDB [database name] to double check
>
|||Thanks Rusell and Sean,
I have changed the model to "BULK_LOGGED" and used -b attribute in bcp import.
Smaller the batch size, faster is the data transfer
"Russell Fields" wrote:
> Shriharsha,
> When BCPing in so much data, it is also good to use the batch size operator
> to control the transaction size. Such as:
> bcp ... -b 50000
> This will break up your bcp into about 300 batches, which will speed it up
> and gives you more transaction log control. You could then (if necessary)
> run extra BACKUP LOGs during the bcp in.
> RLF
> "Sean" <ColdFusion244@.gmail.com> wrote in message
> news:ba587a67-ef18-452e-88a2-d6132234fc5c@.t54g2000hsg.googlegroups.com...
>
>
I am having new issue again,
My task is to copy data from one table to another table residing in
different database.
Table happenes to be extreamly large. (Contains around 15 million
rows.)
I tried several ways (SQL query, SSIS packages,etc...)
I found BCP utility suits my requirement. So planned for BCP.
I am trying following ways (Two steps)
1. bcp <MyFirstDB.TableName> out <MyFlatFilePath> -n -T
2.bcp <MySecondDB.TableName> in <MyFlatFilePath> -n -T
(Basically, Copying data from source to flat file and from there to
DestinationTable.)
Here problem is, My transaction Log file (MySecondDB_log.ldf) grows
like hell on second command. It grows upto 8 GB (max. free space That
I have on disk
My datatransfer will be incomplete because of no space on HardDisk.
PLease let me know, where I am going wrong,if you have better method,
how can i optimize my data transfer. (My log file grows by 2 MB, not
with %ge)
Thank you in advance,
Sriharsha Karagodu.
Are you changing the second database's recovery to bulk?
On Mar 17, 9:56Xam, sriharsha.karag...@.gmail.com wrote:
> Hi Experts,
> I am having new issue again,
> My task is to copy data from one table to another table residing in
> different database.
> Table happenes to be extreamly large. (Contains around 15 million
> rows.)
> I tried several ways (SQL query, SSIS packages,etc...)
> I found BCP utility suits my requirement. So planned for BCP.
> I am trying following ways (Two steps)
> 1. bcp X<MyFirstDB.TableName> out <MyFlatFilePath> -n -T
> 2.bcp X<MySecondDB.TableName> in <MyFlatFilePath> -n -T
> (Basically, Copying data from source to flat file and from there to
> DestinationTable.)
> Here problem is, My transaction Log file (MySecondDB_log.ldf) grows
> like hell on second command. It grows upto 8 GB (max. free space That
> I have on disk
> My datatransfer will be incomplete because of no space on HardDisk.
> PLease let me know, where I am going wrong,if you have better method,
> how can i optimize my data transfer. (My log file grows by 2 MB, not
> with %ge)
> Thank you in advance,
> Sriharsha Karagodu.
|||On Mar 17, 6:59Xpm, Sean <ColdFusion...@.gmail.com> wrote:
> Are you changing the second database's recovery to bulk?
> On Mar 17, 9:56Xam, sriharsha.karag...@.gmail.com wrote:
>
>
>
>
>
>
> - Show quoted text -
Actually I read about Changing the Recovery Property. But Where Do I
Get that option?
When I Do Property of Database--> options-->recovery, This will have
three options like,
TonrnPageDetection,
CheckSum,
None.
So, Not sure where will get option to change the recovery to BULK.
please guide me.
|||You can do it visually, but the syntax goes like this:
ALTER DATABASE [database name]
SET RECOVERY [either: FULL | BULK_LOGGED | SIMPLE]
Also, to see what the current recovery model is run: SP_HELPDB
[database name]
1. Run SP_HELPDB [database name], and note the model used.
2. ALTER DATABASE [database name] SET RECOVERY BULK_LOGGED
3. Run SP_HELPDB [database name] to double check the settings
4. Run bcp import
5. ALTER DATABASE [databasename] SET RECOVERY (output of step 2)
6. SP_HELPDB [database name] to double check
|||Shriharsha,
When BCPing in so much data, it is also good to use the batch size operator
to control the transaction size. Such as:
bcp ... -b 50000
This will break up your bcp into about 300 batches, which will speed it up
and gives you more transaction log control. You could then (if necessary)
run extra BACKUP LOGs during the bcp in.
RLF
"Sean" <ColdFusion244@.gmail.com> wrote in message
news:ba587a67-ef18-452e-88a2-d6132234fc5c@.t54g2000hsg.googlegroups.com...
> You can do it visually, but the syntax goes like this:
> ALTER DATABASE [database name]
> SET RECOVERY [either: FULL | BULK_LOGGED | SIMPLE]
> Also, to see what the current recovery model is run: SP_HELPDB
> [database name]
> 1. Run SP_HELPDB [database name], and note the model used.
> 2. ALTER DATABASE [database name] SET RECOVERY BULK_LOGGED
> 3. Run SP_HELPDB [database name] to double check the settings
> 4. Run bcp import
> 5. ALTER DATABASE [databasename] SET RECOVERY (output of step 2)
> 6. SP_HELPDB [database name] to double check
>
|||Thanks Rusell and Sean,
I have changed the model to "BULK_LOGGED" and used -b attribute in bcp import.
Smaller the batch size, faster is the data transfer
"Russell Fields" wrote:
> Shriharsha,
> When BCPing in so much data, it is also good to use the batch size operator
> to control the transaction size. Such as:
> bcp ... -b 50000
> This will break up your bcp into about 300 batches, which will speed it up
> and gives you more transaction log control. You could then (if necessary)
> run extra BACKUP LOGs during the bcp in.
> RLF
> "Sean" <ColdFusion244@.gmail.com> wrote in message
> news:ba587a67-ef18-452e-88a2-d6132234fc5c@.t54g2000hsg.googlegroups.com...
>
>
bcp - Transaction LogFile Size issue
Hi Experts,
I am having new issue again,
My task is to copy data from one table to another table residing in
different database.
Table happenes to be extreamly large. (Contains around 15 million
rows.)
I tried several ways (SQL query, SSIS packages,etc...)
I found BCP utility suits my requirement. So planned for BCP.
I am trying following ways (Two steps)
1. bcp <MyFirstDB.TableName> out <MyFlatFilePath> -n -T
2.bcp <MySecondDB.TableName> in <MyFlatFilePath> -n -T
(Basically, Copying data from source to flat file and from there to
DestinationTable.)
Here problem is, My transaction Log file (MySecondDB_log.ldf) grows
like hell on second command. It grows upto 8 GB (max. free space That
I have on disk :) ).
My datatransfer will be incomplete because of no space on HardDisk.
PLease let me know, where I am going wrong,if you have better method,
how can i optimize my data transfer. (My log file grows by 2 MB, not
with %ge)
Thank you in advance,
Sriharsha Karagodu.Are you changing the second database's recovery to bulk?
On Mar 17, 9:56=A0am, sriharsha.karag...@.gmail.com wrote:
> Hi Experts,
> I am having new issue again,
> My task is to copy data from one table to another table residing in
> different database.
> Table happenes to be extreamly large. (Contains around 15 million
> rows.)
> I tried several ways (SQL query, SSIS packages,etc...)
> I found BCP utility suits my requirement. So planned for BCP.
> I am trying following ways (Two steps)
> 1. bcp =A0<MyFirstDB.TableName> out <MyFlatFilePath> -n -T
> 2.bcp =A0<MySecondDB.TableName> in <MyFlatFilePath> -n -T
> (Basically, Copying data from source to flat file and from there to
> DestinationTable.)
> Here problem is, My transaction Log file (MySecondDB_log.ldf) grows
> like hell on second command. It grows upto 8 GB (max. free space That
> I have on disk :) ).
> My datatransfer will be incomplete because of no space on HardDisk.
> PLease let me know, where I am going wrong,if you have better method,
> how can i optimize my data transfer. (My log file grows by 2 MB, not
> with %ge)
> Thank you in advance,
> Sriharsha Karagodu.|||On Mar 17, 6:59=A0pm, Sean <ColdFusion...@.gmail.com> wrote:
> Are you changing the second database's recovery to bulk?
> On Mar 17, 9:56=A0am, sriharsha.karag...@.gmail.com wrote:
>
> > Hi Experts,
> > I am having new issue again,
> > My task is to copy data from one table to another table residing in
> > different database.
> > Table happenes to be extreamly large. (Contains around 15 million
> > rows.)
> > I tried several ways (SQL query, SSIS packages,etc...)
> > I found BCP utility suits my requirement. So planned for BCP.
> > I am trying following ways (Two steps)
> > 1. bcp =A0<MyFirstDB.TableName> out <MyFlatFilePath> -n -T
> > 2.bcp =A0<MySecondDB.TableName> in <MyFlatFilePath> -n -T
> > (Basically, Copying data from source to flat file and from there to
> > DestinationTable.)
> > Here problem is, My transaction Log file (MySecondDB_log.ldf) grows
> > like hell on second command. It grows upto 8 GB (max. free space That
> > I have on disk :) ).
> > My datatransfer will be incomplete because of no space on HardDisk.
> > PLease let me know, where I am going wrong,if you have better method,
> > how can i optimize my data transfer. (My log file grows by 2 MB, not
> > with %ge)
> > Thank you in advance,
> > Sriharsha Karagodu.- Hide quoted text -
> - Show quoted text -
Actually I read about Changing the Recovery Property. But Where Do I
Get that option?
When I Do Property of Database--> options-->recovery, This will have
three options like,
TonrnPageDetection,
CheckSum,
None.
So, Not sure where will get option to change the recovery to BULK.
please guide me.|||You can do it visually, but the syntax goes like this:
ALTER DATABASE [database name]
SET RECOVERY [either: FULL | BULK_LOGGED | SIMPLE]
Also, to see what the current recovery model is run: SP_HELPDB
[database name]
1. Run SP_HELPDB [database name], and note the model used.
2. ALTER DATABASE [database name] SET RECOVERY BULK_LOGGED
3. Run SP_HELPDB [database name] to double check the settings
4. Run bcp import
5. ALTER DATABASE [databasename] SET RECOVERY (output of step 2)
6. SP_HELPDB [database name] to double check|||Shriharsha,
When BCPing in so much data, it is also good to use the batch size operator
to control the transaction size. Such as:
bcp ... -b 50000
This will break up your bcp into about 300 batches, which will speed it up
and gives you more transaction log control. You could then (if necessary)
run extra BACKUP LOGs during the bcp in.
RLF
"Sean" <ColdFusion244@.gmail.com> wrote in message
news:ba587a67-ef18-452e-88a2-d6132234fc5c@.t54g2000hsg.googlegroups.com...
> You can do it visually, but the syntax goes like this:
> ALTER DATABASE [database name]
> SET RECOVERY [either: FULL | BULK_LOGGED | SIMPLE]
> Also, to see what the current recovery model is run: SP_HELPDB
> [database name]
> 1. Run SP_HELPDB [database name], and note the model used.
> 2. ALTER DATABASE [database name] SET RECOVERY BULK_LOGGED
> 3. Run SP_HELPDB [database name] to double check the settings
> 4. Run bcp import
> 5. ALTER DATABASE [databasename] SET RECOVERY (output of step 2)
> 6. SP_HELPDB [database name] to double check
>|||Thanks Rusell and Sean,
I have changed the model to "BULK_LOGGED" and used -b attribute in bcp import.
Smaller the batch size, faster is the data transfer
"Russell Fields" wrote:
> Shriharsha,
> When BCPing in so much data, it is also good to use the batch size operator
> to control the transaction size. Such as:
> bcp ... -b 50000
> This will break up your bcp into about 300 batches, which will speed it up
> and gives you more transaction log control. You could then (if necessary)
> run extra BACKUP LOGs during the bcp in.
> RLF
> "Sean" <ColdFusion244@.gmail.com> wrote in message
> news:ba587a67-ef18-452e-88a2-d6132234fc5c@.t54g2000hsg.googlegroups.com...
> > You can do it visually, but the syntax goes like this:
> >
> > ALTER DATABASE [database name]
> > SET RECOVERY [either: FULL | BULK_LOGGED | SIMPLE]
> >
> > Also, to see what the current recovery model is run: SP_HELPDB
> > [database name]
> >
> > 1. Run SP_HELPDB [database name], and note the model used.
> > 2. ALTER DATABASE [database name] SET RECOVERY BULK_LOGGED
> > 3. Run SP_HELPDB [database name] to double check the settings
> > 4. Run bcp import
> > 5. ALTER DATABASE [databasename] SET RECOVERY (output of step 2)
> > 6. SP_HELPDB [database name] to double check
> >
>
>
I am having new issue again,
My task is to copy data from one table to another table residing in
different database.
Table happenes to be extreamly large. (Contains around 15 million
rows.)
I tried several ways (SQL query, SSIS packages,etc...)
I found BCP utility suits my requirement. So planned for BCP.
I am trying following ways (Two steps)
1. bcp <MyFirstDB.TableName> out <MyFlatFilePath> -n -T
2.bcp <MySecondDB.TableName> in <MyFlatFilePath> -n -T
(Basically, Copying data from source to flat file and from there to
DestinationTable.)
Here problem is, My transaction Log file (MySecondDB_log.ldf) grows
like hell on second command. It grows upto 8 GB (max. free space That
I have on disk :) ).
My datatransfer will be incomplete because of no space on HardDisk.
PLease let me know, where I am going wrong,if you have better method,
how can i optimize my data transfer. (My log file grows by 2 MB, not
with %ge)
Thank you in advance,
Sriharsha Karagodu.Are you changing the second database's recovery to bulk?
On Mar 17, 9:56=A0am, sriharsha.karag...@.gmail.com wrote:
> Hi Experts,
> I am having new issue again,
> My task is to copy data from one table to another table residing in
> different database.
> Table happenes to be extreamly large. (Contains around 15 million
> rows.)
> I tried several ways (SQL query, SSIS packages,etc...)
> I found BCP utility suits my requirement. So planned for BCP.
> I am trying following ways (Two steps)
> 1. bcp =A0<MyFirstDB.TableName> out <MyFlatFilePath> -n -T
> 2.bcp =A0<MySecondDB.TableName> in <MyFlatFilePath> -n -T
> (Basically, Copying data from source to flat file and from there to
> DestinationTable.)
> Here problem is, My transaction Log file (MySecondDB_log.ldf) grows
> like hell on second command. It grows upto 8 GB (max. free space That
> I have on disk :) ).
> My datatransfer will be incomplete because of no space on HardDisk.
> PLease let me know, where I am going wrong,if you have better method,
> how can i optimize my data transfer. (My log file grows by 2 MB, not
> with %ge)
> Thank you in advance,
> Sriharsha Karagodu.|||On Mar 17, 6:59=A0pm, Sean <ColdFusion...@.gmail.com> wrote:
> Are you changing the second database's recovery to bulk?
> On Mar 17, 9:56=A0am, sriharsha.karag...@.gmail.com wrote:
>
> > Hi Experts,
> > I am having new issue again,
> > My task is to copy data from one table to another table residing in
> > different database.
> > Table happenes to be extreamly large. (Contains around 15 million
> > rows.)
> > I tried several ways (SQL query, SSIS packages,etc...)
> > I found BCP utility suits my requirement. So planned for BCP.
> > I am trying following ways (Two steps)
> > 1. bcp =A0<MyFirstDB.TableName> out <MyFlatFilePath> -n -T
> > 2.bcp =A0<MySecondDB.TableName> in <MyFlatFilePath> -n -T
> > (Basically, Copying data from source to flat file and from there to
> > DestinationTable.)
> > Here problem is, My transaction Log file (MySecondDB_log.ldf) grows
> > like hell on second command. It grows upto 8 GB (max. free space That
> > I have on disk :) ).
> > My datatransfer will be incomplete because of no space on HardDisk.
> > PLease let me know, where I am going wrong,if you have better method,
> > how can i optimize my data transfer. (My log file grows by 2 MB, not
> > with %ge)
> > Thank you in advance,
> > Sriharsha Karagodu.- Hide quoted text -
> - Show quoted text -
Actually I read about Changing the Recovery Property. But Where Do I
Get that option?
When I Do Property of Database--> options-->recovery, This will have
three options like,
TonrnPageDetection,
CheckSum,
None.
So, Not sure where will get option to change the recovery to BULK.
please guide me.|||You can do it visually, but the syntax goes like this:
ALTER DATABASE [database name]
SET RECOVERY [either: FULL | BULK_LOGGED | SIMPLE]
Also, to see what the current recovery model is run: SP_HELPDB
[database name]
1. Run SP_HELPDB [database name], and note the model used.
2. ALTER DATABASE [database name] SET RECOVERY BULK_LOGGED
3. Run SP_HELPDB [database name] to double check the settings
4. Run bcp import
5. ALTER DATABASE [databasename] SET RECOVERY (output of step 2)
6. SP_HELPDB [database name] to double check|||Shriharsha,
When BCPing in so much data, it is also good to use the batch size operator
to control the transaction size. Such as:
bcp ... -b 50000
This will break up your bcp into about 300 batches, which will speed it up
and gives you more transaction log control. You could then (if necessary)
run extra BACKUP LOGs during the bcp in.
RLF
"Sean" <ColdFusion244@.gmail.com> wrote in message
news:ba587a67-ef18-452e-88a2-d6132234fc5c@.t54g2000hsg.googlegroups.com...
> You can do it visually, but the syntax goes like this:
> ALTER DATABASE [database name]
> SET RECOVERY [either: FULL | BULK_LOGGED | SIMPLE]
> Also, to see what the current recovery model is run: SP_HELPDB
> [database name]
> 1. Run SP_HELPDB [database name], and note the model used.
> 2. ALTER DATABASE [database name] SET RECOVERY BULK_LOGGED
> 3. Run SP_HELPDB [database name] to double check the settings
> 4. Run bcp import
> 5. ALTER DATABASE [databasename] SET RECOVERY (output of step 2)
> 6. SP_HELPDB [database name] to double check
>|||Thanks Rusell and Sean,
I have changed the model to "BULK_LOGGED" and used -b attribute in bcp import.
Smaller the batch size, faster is the data transfer
"Russell Fields" wrote:
> Shriharsha,
> When BCPing in so much data, it is also good to use the batch size operator
> to control the transaction size. Such as:
> bcp ... -b 50000
> This will break up your bcp into about 300 batches, which will speed it up
> and gives you more transaction log control. You could then (if necessary)
> run extra BACKUP LOGs during the bcp in.
> RLF
> "Sean" <ColdFusion244@.gmail.com> wrote in message
> news:ba587a67-ef18-452e-88a2-d6132234fc5c@.t54g2000hsg.googlegroups.com...
> > You can do it visually, but the syntax goes like this:
> >
> > ALTER DATABASE [database name]
> > SET RECOVERY [either: FULL | BULK_LOGGED | SIMPLE]
> >
> > Also, to see what the current recovery model is run: SP_HELPDB
> > [database name]
> >
> > 1. Run SP_HELPDB [database name], and note the model used.
> > 2. ALTER DATABASE [database name] SET RECOVERY BULK_LOGGED
> > 3. Run SP_HELPDB [database name] to double check the settings
> > 4. Run bcp import
> > 5. ALTER DATABASE [databasename] SET RECOVERY (output of step 2)
> > 6. SP_HELPDB [database name] to double check
> >
>
>
2012年2月16日星期四
batching an update, use TRANSACTION?
I have a storedproc that does several UPDATEs to a table, some of which fire
and some of which don't based on various IFs through the proc. There's a
maximum f three, it's not THAT complex.
However there is a trigger on the table that records all UPDATEs into a
separate auditing log. I would like there to be only one trigger fire
regardless if one, two or three of the UPDATEs were called.
Does TRANSACTION do this? I do _not_ want to turn off the triggers in the
proc.
MauryIf I understand you correctly, TRANSACTION will not do it. Can you put in
some logic in your trigger to check whether a previous UPDATE has already
fired the trigger?
Linchi
"Maury Markowitz" wrote:
> I have a storedproc that does several UPDATEs to a table, some of which fire
> and some of which don't based on various IFs through the proc. There's a
> maximum f three, it's not THAT complex.
> However there is a trigger on the table that records all UPDATEs into a
> separate auditing log. I would like there to be only one trigger fire
> regardless if one, two or three of the UPDATEs were called.
> Does TRANSACTION do this? I do _not_ want to turn off the triggers in the
> proc.
> Maury|||"Linchi Shea" wrote:
> If I understand you correctly, TRANSACTION will not do it. Can you put in
> some logic in your trigger to check whether a previous UPDATE has already
> fired the trigger?
I guess, but only with peril.
Maury|||"Linchi Shea" wrote:
> If I understand you correctly, TRANSACTION will not do it. Can you put in
> some logic in your trigger to check whether a previous UPDATE has already
> fired the trigger?
Can I perhaps wrap the individual fields of the update in some sort of
conditional, and thereby combine them into one larger statement? Everything
is already loaded into local vars.
Maury
and some of which don't based on various IFs through the proc. There's a
maximum f three, it's not THAT complex.
However there is a trigger on the table that records all UPDATEs into a
separate auditing log. I would like there to be only one trigger fire
regardless if one, two or three of the UPDATEs were called.
Does TRANSACTION do this? I do _not_ want to turn off the triggers in the
proc.
MauryIf I understand you correctly, TRANSACTION will not do it. Can you put in
some logic in your trigger to check whether a previous UPDATE has already
fired the trigger?
Linchi
"Maury Markowitz" wrote:
> I have a storedproc that does several UPDATEs to a table, some of which fire
> and some of which don't based on various IFs through the proc. There's a
> maximum f three, it's not THAT complex.
> However there is a trigger on the table that records all UPDATEs into a
> separate auditing log. I would like there to be only one trigger fire
> regardless if one, two or three of the UPDATEs were called.
> Does TRANSACTION do this? I do _not_ want to turn off the triggers in the
> proc.
> Maury|||"Linchi Shea" wrote:
> If I understand you correctly, TRANSACTION will not do it. Can you put in
> some logic in your trigger to check whether a previous UPDATE has already
> fired the trigger?
I guess, but only with peril.
Maury|||"Linchi Shea" wrote:
> If I understand you correctly, TRANSACTION will not do it. Can you put in
> some logic in your trigger to check whether a previous UPDATE has already
> fired the trigger?
Can I perhaps wrap the individual fields of the update in some sort of
conditional, and thereby combine them into one larger statement? Everything
is already loaded into local vars.
Maury
2012年2月13日星期一
Batch Insert into a Table
I have a large volume of records I need to insert into a table, but am
running into issues with my transaction log filling up. I want to
create a commit point at every 50-100k records, but don't know how to
do that in SQL Server. I was thinking it has something to do with the
@.@.TRANCOUNT or @.@.ROWCOUNT system variables, but again, not sure how to
use them.
Does anyone have any code or suggestions on an approach?
Many thanks,
DanYou're transaction log will fill up when doing large inserts, regardless of
whether you do it in small batches or all at once.
You could use ALTER DATABASE to put your db into bulked-logged recovery
mode, then do the insert
ALTER DATABASE [YourDb] SET RECOVERY BULK_LOGGED
--do large insert, then set recovery model back to full
ALTER DATABASE [YourDb] SET RECOVERY FULL
"Dan" wrote:
> I have a large volume of records I need to insert into a table, but am
> running into issues with my transaction log filling up. I want to
> create a commit point at every 50-100k records, but don't know how to
> do that in SQL Server. I was thinking it has something to do with the
> @.@.TRANCOUNT or @.@.ROWCOUNT system variables, but again, not sure how to
> use them.
> Does anyone have any code or suggestions on an approach?
> Many thanks,
> Dan
>|||I should also note that the data is already in the database and I'm
trying to perform an insert from a staging/temp table into another
table:
insert into final_table (field1, field2,...fieldN)
select field1, field2,...fieldN
from staging_table
running into issues with my transaction log filling up. I want to
create a commit point at every 50-100k records, but don't know how to
do that in SQL Server. I was thinking it has something to do with the
@.@.TRANCOUNT or @.@.ROWCOUNT system variables, but again, not sure how to
use them.
Does anyone have any code or suggestions on an approach?
Many thanks,
DanYou're transaction log will fill up when doing large inserts, regardless of
whether you do it in small batches or all at once.
You could use ALTER DATABASE to put your db into bulked-logged recovery
mode, then do the insert
ALTER DATABASE [YourDb] SET RECOVERY BULK_LOGGED
--do large insert, then set recovery model back to full
ALTER DATABASE [YourDb] SET RECOVERY FULL
"Dan" wrote:
> I have a large volume of records I need to insert into a table, but am
> running into issues with my transaction log filling up. I want to
> create a commit point at every 50-100k records, but don't know how to
> do that in SQL Server. I was thinking it has something to do with the
> @.@.TRANCOUNT or @.@.ROWCOUNT system variables, but again, not sure how to
> use them.
> Does anyone have any code or suggestions on an approach?
> Many thanks,
> Dan
>|||I should also note that the data is already in the database and I'm
trying to perform an insert from a staging/temp table into another
table:
insert into final_table (field1, field2,...fieldN)
select field1, field2,...fieldN
from staging_table
2012年2月11日星期六
basic tlog question
whenever a transaction or a SQL statement executes, do the changes get
written to the log file first ? I know they do, but are they also in
memory.. I understand the part where dirty pages are written to disk.. at
checkpoints or lazy writers or thru worker threads...what im a bit confused
is when it talks about writing to disk, is it referring to the data files on
disk or the log files on disk..
Can someone just give me a 2 to 3 liner on the initial part before the data
actually gets written to the disk i.e. disk that contains the data files ?
Yes, the log records are written to memory too, to an area called 'log
cache', and SQL Server has a logic to make sure these cached log records are
written to log files, before the associated dirty pages are written to disk.
Vyas, MVP (SQL Server)
http://vyaskn.tripod.com/
"Hassan" <fatima_ja@.hotmail.com> wrote in message
news:e3WwGOvaEHA.3524@.TK2MSFTNGP12.phx.gbl...
> whenever a transaction or a SQL statement executes, do the changes get
> written to the log file first ? I know they do, but are they also in
> memory.. I understand the part where dirty pages are written to disk.. at
> checkpoints or lazy writers or thru worker threads...what im a bit
confused
> is when it talks about writing to disk, is it referring to the data files
on
> disk or the log files on disk..
> Can someone just give me a 2 to 3 liner on the initial part before the
data
> actually gets written to the disk i.e. disk that contains the data files ?
>
written to the log file first ? I know they do, but are they also in
memory.. I understand the part where dirty pages are written to disk.. at
checkpoints or lazy writers or thru worker threads...what im a bit confused
is when it talks about writing to disk, is it referring to the data files on
disk or the log files on disk..
Can someone just give me a 2 to 3 liner on the initial part before the data
actually gets written to the disk i.e. disk that contains the data files ?
Yes, the log records are written to memory too, to an area called 'log
cache', and SQL Server has a logic to make sure these cached log records are
written to log files, before the associated dirty pages are written to disk.
Vyas, MVP (SQL Server)
http://vyaskn.tripod.com/
"Hassan" <fatima_ja@.hotmail.com> wrote in message
news:e3WwGOvaEHA.3524@.TK2MSFTNGP12.phx.gbl...
> whenever a transaction or a SQL statement executes, do the changes get
> written to the log file first ? I know they do, but are they also in
> memory.. I understand the part where dirty pages are written to disk.. at
> checkpoints or lazy writers or thru worker threads...what im a bit
confused
> is when it talks about writing to disk, is it referring to the data files
on
> disk or the log files on disk..
> Can someone just give me a 2 to 3 liner on the initial part before the
data
> actually gets written to the disk i.e. disk that contains the data files ?
>
basic tlog question
whenever a transaction or a SQL statement executes, do the changes get
written to the log file first ? I know they do, but are they also in
memory.. I understand the part where dirty pages are written to disk.. at
checkpoints or lazy writers or thru worker threads...what im a bit confused
is when it talks about writing to disk, is it referring to the data files on
disk or the log files on disk..
Can someone just give me a 2 to 3 liner on the initial part before the data
actually gets written to the disk i.e. disk that contains the data files ?Yes, the log records are written to memory too, to an area called 'log
cache', and SQL Server has a logic to make sure these cached log records are
written to log files, before the associated dirty pages are written to disk.
--
Vyas, MVP (SQL Server)
http://vyaskn.tripod.com/
"Hassan" <fatima_ja@.hotmail.com> wrote in message
news:e3WwGOvaEHA.3524@.TK2MSFTNGP12.phx.gbl...
> whenever a transaction or a SQL statement executes, do the changes get
> written to the log file first ? I know they do, but are they also in
> memory.. I understand the part where dirty pages are written to disk.. at
> checkpoints or lazy writers or thru worker threads...what im a bit
confused
> is when it talks about writing to disk, is it referring to the data files
on
> disk or the log files on disk..
> Can someone just give me a 2 to 3 liner on the initial part before the
data
> actually gets written to the disk i.e. disk that contains the data files ?
>
written to the log file first ? I know they do, but are they also in
memory.. I understand the part where dirty pages are written to disk.. at
checkpoints or lazy writers or thru worker threads...what im a bit confused
is when it talks about writing to disk, is it referring to the data files on
disk or the log files on disk..
Can someone just give me a 2 to 3 liner on the initial part before the data
actually gets written to the disk i.e. disk that contains the data files ?Yes, the log records are written to memory too, to an area called 'log
cache', and SQL Server has a logic to make sure these cached log records are
written to log files, before the associated dirty pages are written to disk.
--
Vyas, MVP (SQL Server)
http://vyaskn.tripod.com/
"Hassan" <fatima_ja@.hotmail.com> wrote in message
news:e3WwGOvaEHA.3524@.TK2MSFTNGP12.phx.gbl...
> whenever a transaction or a SQL statement executes, do the changes get
> written to the log file first ? I know they do, but are they also in
> memory.. I understand the part where dirty pages are written to disk.. at
> checkpoints or lazy writers or thru worker threads...what im a bit
confused
> is when it talks about writing to disk, is it referring to the data files
on
> disk or the log files on disk..
> Can someone just give me a 2 to 3 liner on the initial part before the
data
> actually gets written to the disk i.e. disk that contains the data files ?
>
basic tlog question
whenever a transaction or a SQL statement executes, do the changes get
written to the log file first ? I know they do, but are they also in
memory.. I understand the part where dirty pages are written to disk.. at
checkpoints or lazy writers or thru worker threads...what im a bit confused
is when it talks about writing to disk, is it referring to the data files on
disk or the log files on disk..
Can someone just give me a 2 to 3 liner on the initial part before the data
actually gets written to the disk i.e. disk that contains the data files ?Yes, the log records are written to memory too, to an area called 'log
cache', and SQL Server has a logic to make sure these cached log records are
written to log files, before the associated dirty pages are written to disk.
--
Vyas, MVP (SQL Server)
http://vyaskn.tripod.com/
"Hassan" <fatima_ja@.hotmail.com> wrote in message
news:e3WwGOvaEHA.3524@.TK2MSFTNGP12.phx.gbl...
> whenever a transaction or a SQL statement executes, do the changes get
> written to the log file first ? I know they do, but are they also in
> memory.. I understand the part where dirty pages are written to disk.. at
> checkpoints or lazy writers or thru worker threads...what im a bit
confused
> is when it talks about writing to disk, is it referring to the data files
on
> disk or the log files on disk..
> Can someone just give me a 2 to 3 liner on the initial part before the
data
> actually gets written to the disk i.e. disk that contains the data files ?
>
written to the log file first ? I know they do, but are they also in
memory.. I understand the part where dirty pages are written to disk.. at
checkpoints or lazy writers or thru worker threads...what im a bit confused
is when it talks about writing to disk, is it referring to the data files on
disk or the log files on disk..
Can someone just give me a 2 to 3 liner on the initial part before the data
actually gets written to the disk i.e. disk that contains the data files ?Yes, the log records are written to memory too, to an area called 'log
cache', and SQL Server has a logic to make sure these cached log records are
written to log files, before the associated dirty pages are written to disk.
--
Vyas, MVP (SQL Server)
http://vyaskn.tripod.com/
"Hassan" <fatima_ja@.hotmail.com> wrote in message
news:e3WwGOvaEHA.3524@.TK2MSFTNGP12.phx.gbl...
> whenever a transaction or a SQL statement executes, do the changes get
> written to the log file first ? I know they do, but are they also in
> memory.. I understand the part where dirty pages are written to disk.. at
> checkpoints or lazy writers or thru worker threads...what im a bit
confused
> is when it talks about writing to disk, is it referring to the data files
on
> disk or the log files on disk..
> Can someone just give me a 2 to 3 liner on the initial part before the
data
> actually gets written to the disk i.e. disk that contains the data files ?
>
订阅:
博文 (Atom)