I have a SP which have 10 insert SQL commands coming from same stage table but differs at joins which executes one after the other but it is taking too much time to execute and all the results should be going to only one destination table. So i made each insert command with different table joins into one each SP so that i thought i can execute them parllel and improves the performance. But unfortunately the performance has become worst, reason is that i'm trying to insert from multiple SP's onto one table causing the issue more. So basically how can i handle this kind of situation. FYI i have all indexes on all the tables which i'm using currently. Since it is execute process task i cannot set **rows per batch option** How can i improve the insert performance on destination table
↧
How can i improve insert performance on a same SQL table if you have multiple sp's which inserts rows
↧
How to load data from multiple tables to multiple tables by using SSIS
I am using SQL Server 2008 R2. I have almost 106+ tables in database and now I need to load data from these tables into another database. Table structure of source and destination database are same. There is a status table in the database and all tables have a Primary Key ID field. All the tables have Primary key and Foreign key relationship. I need to transfer data those ID number will match with Status table where ID status is 1 or something like this. I need to insert Identity column also, and because of all the table have PK and FK relationship so i have to insert data into tables with referential constraints.
So, my questions here is: To fulfill my above scenario how can I load data from multiple tables to multiple tables by using SSIS because it is not one time thing.
↧
↧
Create a dynamic solution with SSIS
Hello all,
I need to create a process in SSIS that will do the following things per each csv file separately. Each file has a different number of columns and different data types, so I need to come up with a generic solution. So, in a nutshell:
1. Check each file's number of columns and data type.
2. Apply sanity checks to each column. For example, integer should not have (.).
3. As a final outcome, the file will contain all the rows and two additional columns, errorcolumns and errordescription for example:
Column1 Column2 Column3 Column4 Column5 Column6 ErrorColumns ErrorDescription
500 800 Test1 Test2 200.8 1/6/2017
80 Test3 Test8 10,2 Column2|Column3|Column4|Column5|Column6 Null|not int|not int|not float|null
Could you please help me?
Thank you in advance.
↧
How to load data from multiple tables to multiple tables by using SSIS
I am using SQL Server 2008 R2. I have almost 106+ tables in database and now I need to load data from these tables into another database. Table structure of source and destination database are same. There is a status table in the database and all tables have a Primary Key ID field. All the tables have Primary key and Foreign key relationship. I need to transfer data those ID number will match with Status table where ID status is 1 or something like this. I need to insert Identity column also, and because of all the table have PK and FK relationship so i have to insert data into tables with referential constraints.
So, my questions here is: To fulfill my above scenario how can I load data from multiple tables to multiple tables by using SSIS because it is not one time thing.
↧
SSIS Package fails when running with SQL Agent
Hello,
I am having a problem with an SSIS package that I am running with a scheduled task with SQL Agent. If I run the package manually with SQL Server Information Services it works, but when I automate the task with SQL agent it fails. I am using SQL Server 2008.
I created a VM to replicate my server in production with the same SQL Server version and the task with SQL agent does not fail, but in the other server fails. The server in production is running other tasks with the agent that runs successfully, but this new task fails.
Attached is a sample of the data the SSIS pkg is converting to fill out 4 tables: Employees, OU, EmployeeGroup, and Groups.
I also attached the error that I am getting.
I can figure out why the pkg runs successfully if I run it manually, but fails when is automated to run with the SQL agent.
To clarify, Jordan, the flat file I am using is the same to run the job in the VM is the same I am using in the server in production.
Also, the account I am using to run the package has access to the flat file, and I am using a Proxy account to run the package.
This is the error I get:
Message
Executed as user: abc. Microsoft (R) SQL Server Execute Package Utility Version 10.50.6000.34 for 64-bit Copyright (C) Microsoft Corporation 2010. All rights reserved. Started: 11:48:05 AM Error: 2017-06-29 11:48:07.97 Code: 0xC020209C Source: DFT Employee Table Flat File Source [628] Description: The column data for column "UserName" overflowed the disk I/O buffer. End Error Error: 2017-06-29 11:48:07.97 Code: 0xC0202091 Source: DFT Employee Table Flat File Source [628] Description: An error occurred while skipping data rows. End Error Error: 2017-06-29 11:48:07.97 Code: 0xC0047038 Source: DFT Employee Table SSIS.Pipeline Description: SSIS Error Code DTS_E_PRIMEOUTPUTFAILED. The PrimeOutput method on component "Flat File Source" (628) returned error code 0xC0202091. The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing. There may be error messages posted before this with more information about the failure. End Error DTExec: The package execution returned DTSER_FAILURE (1). Started: 11:48:05 AM Finished: 11:48:08 AM Elapsed: 2.028 seconds. The package execution failed. The step failed.
Any suggestions on this issue?
Thanks.
↧
↧
ActiveX Script errors in SSIS package
Hi everyone,
I have a SSIS package that ia ran manually everything month end for reporting.
It ran successfully in last month but this month it is giving an error message below;
"Error: 0xC00291B1 at Populate And Copy Temp DB, ActiveX Script Task: User script threw an exception: "Error Code: 0 Error Source= ADODB.Connection Error Description: Provider cannot be found. It may not be properly installed
"
Please, can someone help to figure out how this error can be resolved.
Thanks
↧
Character conversion issue in SSIS
Hi All,
I'm trying to import a ANSI format file which contains a name **Portomeñe**. But when I import it using the DAta flow task, it is importing as **Portome�e**. The special character ñ corrupting after the import. The data type of the column where I'm importing the data (whole row) is varchar(7000). Please suggest how to fix this issue. I'm in urgent need of fixing this issue.
Thanks in advance...
↧
How Do I transform text-based rows into columns?
Hello,
I have a table that has the following columns: ProjectID, SequenceNumber, and TextField. This table is used to store project roles. Each role has a unique row. For example, if I have project number 1234, 12 (ProjectManager) as sequence number (which is actually a title), John Doe in the text field and the next row has project number 1234, 13 (ProjectCoordinator) as sequence number, Jane Doe in the text field, and so on. My instinct says that this is a prime candidate for either the pivot function in t-sql or the pivot transformation in SSIS. In t-sql I tried using ROW_NUMBER() OVER GROUP BY () and then max in the pivot function, but I still am unable to get it to work. My output should be:
Project ProjectManager (Seq #12) Project Coordinator (Seq #13)
1234 John Doe Jane Does
I suppose I could do a lookup in SSIS, but that seems rather tedious.
Thanks.
Tom
↧
SSIS Package Export using command lines?
We all know that SSIS projects/packages any of the follow methods:
1. File system
2. MSDB
3. SSIS Catalog
Can anyone help me on getting the dtsx files and config files from MSDB / SSIS Catalog.
If SSIS package is deployed in MSDB/SSIS Catalog, through GUI we can export the packages with the help of Management Studio's Integration Service.
Can anyone help to export SSIS packages into .dtsx / .ispac files through commands (using command prompt)??
↧
↧
How to create a Data Flow Task when using a variable to pass the connection string for that task.
I am storing my connection information for external locations in the local host where the SSIS resides. My first step is to access the local host to get connection information and create a connection string variable. I set the expression on the connection string to the variable. Now I am trying to create the data flow task source to extract data from the external location. I continuously get errors saying I am not connected. I tried setting the connection manager first but since the variable is blank at pre-runtime, the connection manager is not connected. I am thinking I should disable the variable until after I have specified my source and destination for the data flow task, but how do I test that my variable is actually working? Have I confused you yet? In a nutshell, I connect to DB1 to get the connection string for DB2, then extract data from DB2 to DB1. Help???
Thanks!
Paws27284
↧
Documentation tool for SSIS
Hello! I am looking for a tool to document complex SSIS packages.
I researched APEXSQL and Pragmatic Works DOC XPRESS .
Any other good ones?
Thank you
↧
How to Log New Value and updated value in SSIS
I am using Lookup transformation to check the match and unmatch records. For matching records i am doing Update with the SQL update statement in the SQL command .
In my project, My ssis should capture the new and updated value once the update is over. it should be in the below format as a table.
id, Columnname, Old Value , new Value .
Please help me on this,
↧
how to edit connection manager variables
Hi All
There is a package that started to fail. The connection manager has a function sign on it. In the expressions i have two expressions
@[$Package::Database]
@[$Package::Server]
which when i evaluate is showing the wrong server and the wrong database name and hence the package is failing I never worked with variables, can someone please tell me where to go to edit these variables?
Also when i click on the variables icon i do not see these variaables can someone please suggest as to from where they are pushed ?
Thanks
[SSIS][1]
[1]: https://mindmajix.com/ssis-training
↧
↧
SSIS Error - Excel Connection - Class not registered
Connection manager "Excel Connection Manager"
Description: SSIS Error Code DTS_E_OLEDB_NOPROVIDER_ERROR. The requested OLE DB provider Microsoft.ACE.OLEDB.12.0 is not registered.
Error code: 0x00000000. An OLE DB record is available. Source: "Microsoft OLE DB Service Components" Hresult: 0x80040154 Description: "Class not registered".
Trying to read an excel file and getting a connection problem.
The interesting thing is if I go to query analyzer and use sql code to read the excel file I have no issues. It just does not work via SSIS. Any ideas.
↧
fixed length flat file
I am using SSIS to create a Fixed Length flat file for a end user who does not have ability to run SSIS job.
I am looking for a way to allow this user to create this file on demand.
I have a SQL query ( stored Procedure) that I am using along with SSIS.
By the way, everything will work if this was done in SSRS, which essentially does not give me ability to create a FL flat file.
What is the best way for me to make this happen? Thanks.
↧
How to rename external file in SSIS
I have a folder for my destination excel file called MyDest. It has let's say 3 files:
Output.xls
Output_062017.xls
Output_072017.xls
When I'm creating new file for MyDest folder it must be named Output.xls
So what I need is to rename existing Output.xls to Output_082017.xls and after that put new Output.xls into that folder
Please help
↧
I have encounter the following error while running DQS component in SSIS?
HI All ,
I have encoun.ter the following error while running DQS component in SSIS
DFL-Clean Data:Error:
Microsoft.Ssdqs.Infra.Exceptions.EntryPointException:The
attempt to update or delete a DAO object of type
'AKnowledgebase' with id 1001679 has failed because the object is not up to date or is
being deleted from the database
at
Microsoft.Ssdqs.Proxy.Database.DBAcessClient.Exec()
at
Microsoft.Ssdqs.Proxy.EntryPoint.KnowledgebaseManagementEntryPointClient.DQProjectDelete
(DQProjectdqProject)
at
Microsoft.Ssdqs.Component.DataCorrection.Logic.DataCorrectionComponent.ProcessInput
(Int32inputID,PipelineBuffer buffer)
at
Microsoft.SqlServer.Dts.Pipeline.ManagedComponentHost.HostprocessInput(IDTSManagedComponent
Wrapper100 Wrapper,Int32inputID,IDTSBuffer100 pDTSBuffer, IntPtrbufferWirePacket)
DFL-Clean Data:Error: An Unexpected error occured in the DQS infrastructure
system.NullReferenceException:object reference not to an instance of an object
at
Microsoft.Ssdqs.Component.DataCorrection.Logic.DataCorrectionComponent.ProcessChunk
(ReadonlyCollection'1filedmappings,ReadonlyCollection '1 records,
CorrectedRecordsStatusStatistics&
CorrectedRecordsTotalStatusStatistics)
↧
↧
SQL job won't run ssis pkg on VMWARE
I created a Kingsway soft ssis pkg and deployed. I also created a SQL job but it fails step 1.
This solution sits on a VM owned by a client. Could this failure be a result of a missing proxy?
↧
ETL : Slowness
Hi Team,
I am working for Indian NBFC in BIU team.
I am responsible for EDW management and reporting solutions. Currently, one of leading Vendor is supporting us for EDW activities (ETL - SSIS, SSAS, SSRS).
From last few months we are facing ETL performance issue where ETL timing has increased drastically. We have done initial RCA , and found its due to the sudden data volume growth. So we have enhanced our server infra by adding drive space, RAM and Core. In spite of increasing server capacity we are still facing issue with the ETL timing. Also we are not able to address Server bandwidth, vendor’s point is we are not getting bandwidth to fetch data (read / write) but as per IT infra team we have good bandwidth (256 MB) but our SQL services not able to consume the same.
I need help to address two major issues here;
how to crunch down ETL timing further
What are the technical things I need to check
How we will get constant bandwidth for ETL process
Thanks,
Pravin Vithu Kadam
India +91-9987037828
↧
Extract to multiple excel files with dynamic file naming
I currently use a cursor to iterate through a table selecting data to bcp out to various excel files and dynamically name the files as I go. The file names are constructed from data in the database.
All the files have different data, but have exactly the same column structure, and have just 1 worksheet each.
I want to use SSIS to do the export instead. I have got as far as using SSIS to export the data using separate queries/dataflows and hardcoding the file names etc. The problem is that I want to make it dynamic so that it can cope with any number of excel exports and file names.
This page shows brilliantly how to import multiple excel files dynamically, but I want to do the exact opposite!
https://stackoverflow.com/questions/7411741/how-to-loop-through-excel-files-and-load-them-into-a-database-using-ssis-package
I am using VS2015 & sql 2012
Any help that can be offered (with examples?) would be greatly appreciated
↧