WebSphere QualityStage

Replacement stages to select the WebSphere DataStage stage to substitute for the QualityStage 7 stage.

Datastage Runtime Column propagation

Enable the RCP feature, stages in parallel jobs can handle undefined
columns that they encounter when the job is run, and propagate these
columns through to the rest of the job.

Enable Runtime Column Propagation for Parallel Jobs
If you enable this feature, stages in parallel jobs can handle undefined columns that they encounter when the job is run, and propagate these columns through to the rest of the job. This check box enables the feature, to actually use it you need to explicitly select the option on each stage

If runtime column propagation is enabled in the DataStage Administrator, you can select the Runtime column propagation to specify that columns encountered by a stage in a parallel job can be used even if they are not explicitly defined in the meta data. You should always ensure that runtime column propagation is turned on if you want to use schema files to define column meta data.

There are some special considerations when using runtime column propagation with certain stage types:

# Sequential File
# File Set
# External Source
# External Target

RCP Set at DataStage Adminstrator:
RCP Set at DataStage Stage Output:

Merge Stage: Ensure required column meta data has been specified (this may be done in another stage, or may be omitted altogether if you are relying on Runtime Column Propagation).

Lookup Stage: Ensure required column meta data has been specified (this may be omitted altogether if you are relying on Runtime Column Propagation).

Shared Container: When inserted in a job, a shared container instance already has meta data defined for its various links. This meta data must match that on the link that the job uses to connect to the container exactly in all properties. The Inputs page enables you to map meta data as required. The only exception to this is where you are using runtime column propagation (RCP) with a parallel shared container. If RCP is enabled for the job, and specifically for the stage whose output connects to the shared container input, then meta data will be propagated at run time, so there is no need to map it at design time.

For parallel shared containers, you can take advantage of runtime column propagation to avoid the need to map the meta data. If you enable runtime column propagation, then, when the jobs runs, meta data will be automatically propagated across the boundary between the shared container and the stage(s) to which it connects in the job.

Data Warehouse Bus Architecture

ClearCase Commands

ClearCase Commands
-----------------------------------------------------------------

IBM Rational ClearCase is a powerful, elegant version control system.

Since it's heritage is Unix, the manual is ... obscure. These battle tested commands should get you up and running...

Before you dive in take note!

'cl' is the accepted shortform for 'ClearCase' / 'Cleartool' when entering package commands.

A period in a sample command line is not punctuation, it's meant to be typed!

Bon appetit.

Basics
01 To Get Help
02 To enter cleartool.
03 To exit cleartool.
04 To navigate inside cleartool.
05 To enter graphics mode (slow but gorgeous).

File Checkouts and Checkins
06 To Check out a file.
07 To cancel a checkout.
08 To check out a previous version of a file.
09 To find what you have checked out.
10 To find out what other files are checked out.
11 To Check in a file.
12 To checkin or checkout multiple files.

Labels
13 To list existing labels.
14 To observe the naming conventions for branches and labels.
15 To Make a label.
16 To label a version of a file.
17 To apply a label to the entire source tree.
18 To move an existing label down to the latest elements.
19 To see if a label is on the latest version of a file or on a branch.

File and Version Information
20 To change the comment for a revision.
21 To change the content description for a file.
22 To list the history (versions with comments) of a file.
23 To find out what files have been worked on since ....
24 To find out the file_type of a file.
25 To setup cleartool to send email on a file change.

Directory Operations
26 To add a new directory to the source tree.
27 To rename a directory in the /gl/vobs/netman/src tree.
28 To rename a vob directory or element.

File Operations : Add Move Rename
29 To Add a new file to an existing directory.
30 To use symbolic links avoid duplicating an archive file in different directories.
31 To move an atria file from one directory to another.
32 To make a view private copy of a file.
33 To remove 'rm' a Cleartool archive file.
34 To look at a file that has been Cleartool 'rm'd'.

Branches
35 To List existing branches.
36 To Make a branch.
37 To merge a branch back to the mainline.

Releases
38 To build a release.
39 To obtain a directory listing of the source tree for a release.
40 To obtain a list of file changes since the last release.

Administrator
41 To remove (destroy) the latest version(s) of a file.
42 To look at your configspec.
43 To change your configspec.

ClearCase Commands : Basics

01. To get help
DOS bigots will appreciate the intuitive nature of the Unix equivalent of 'Help': 'man' (aka Manual Pages)
To get help on the entire command set enter:
cleartool man
To get help on a specific command enter:
cleartool man command
(eg: cleartool man co for help on the checkout command)
________________________________________

02. To enter Cleartool
cl
The prompt will change to cleartool.
________________________________________

03. To exit Cleartool.
exit
or
quit
________________________________________

04. To navigate inside Cleartool
Use the standard cd or cd .. commands to navigate the directory tree.

Warning! Be aware that if you exit from Cleartool back to the shell after having moved around in the directory tree within Cleartool, that you will NOT be returned to the directory that you were in when you entered Cleartool from the shell.
The pwd (present working directory) command can help you keep track of where you are.
________________________________________
05. To enter graphics mode (slow but gorgeous).
cl xlsvtree filename
or
xcleartool
The xlsvtree graphics viewer is a powerful tool for viewing the version tree of a file. It is very useful for visual diffing of one version of the file against another, labelling versions or moving labels around.


ClearCase Commands : File Checkin Checkout
06. To check out a file.
cleartool checkout file.ext
or
cl co file.ext
cl co -c "comment" file.ext
End comment with: .or
Cleartool will prompt for a comment if you do not use the third format to supply one.
Although you may not be used to providing a description of the changes at checkout time ot is good practice to enter the reason you are checking out the file.
Other team members can then see who has the file out and why.
When you check the file back in this text will be presented as the default ci change description, which you can override. If you are forethoughtful about the comment most often it will not need to be re-typed when you do the checkin.
The file checked out stays in the same directory but the permission changes from read only to read/write.
________________________________________

07. To cancel a checkout.
cl unco file.ext
________________________________________

08. To check out a previous version of a file.
This should be:
cl co file.ext@@/main/version#>
and it is a defect in the tool that this command is not implemented.
Instead, you will have to do a tap dance to get the version you want in your view:
#1. cl co file.ext
#2. cl exit to return to the shell
#3. pwd to make sure you are in the right directory at the shell level. If not, navigate to the right directory.
#4. cp file.ext/main/version# tmpfile.ext
You can then change tmpfile as required, cp/copy it to the atria directory with the correct name, enter cleartool again and check the file in. Exit cleartool to the shell to rm/delete the temporary file. Using a tmp_name will help ensure that you don't screw up and delete the atria archive file by mistake!
________________________________________

09. To find what you have checked out.
cl lsco -me -all
Remember to do this at the end of your session to make sure you have not forgotten to check back in files and directories.
Other team members cannot access the files for update until you have checked them back in, and will have to waste time diffing and merging your changes with theirs if they are forced to use private copies.
________________________________________

10. To find out what other files are checked out.
cl lsco -all
________________________________________

11. To check in a file.
cleartool checkin file.ext
or
cl ci -c "comment" file.ext
Cleartool will display the comment you entered at co time as the default. You may override this if you want to.
End comment with . or -d
Do NOT use -nc which means no comment.
Comment your versions!!!!
________________________________________

12. To checkin or checkout multiple files.
To check in, or check out multiple files at once with the same comment:
cl ci -c "comment" *
cl co -c "comment" *



ClearCase Commands : Labels
13. To list existing labels.
cl lstype -lbtype
________________________________________

14. To observe the naming conventions for branches and labels.
branch names: all lower case eg: forwarding, ip, gap
LABELS : ALL UPPER CASE eg: BETA_2.5, PRE_GAP_MERGE
________________________________________

15. To Make a label.
cl mklbtype BETA_1.1B
Note: The convention for labels is as above: all caps.
________________________________________

16. To label a version of a file.
cl mklabel -c "comment" -rec x.
Warning! Do not confuse these foolishly named operations:
mklbtype makes (creates) a new label.
whereas
mklabel ATTACHES the label made by mklbtype to a specific file.
________________________________________

17. To apply a label to the entire source tree.
This is typically done for alpha, beta, final releases so that the entire set of files with the label can be built.
#1 Use 'catcs/edcs' (Tip 43) to set up your configspec with
element * MyLabel
element * /main/LATEST
#2 mklbtype MyLabel and enter appropriate comments.
#3 Be in the top level (/src) directory.
To apply the label throughout the tree:
mklabel -recurse -c "X Release" MyLabel .
Wait a long time.
________________________________________

18. To move an existing label down to the latest elements.
mklabel -recurse -replace MY_LABEL
will move existing label MY_LABEL down to the latest revision of each file in the directory, and below, from which you invoke this command.
'replace' is a poorly named option: it really means 'move-down'.
________________________________________

19. To see if a label is on the latest version of a file or on a branch.
cl find -version 'lbtype(LABELNAME) && version(/main/LATEST)' -print
or
cl find . -version 'version(/main/LATEST) && ! version(/main/GAP)' -print
You can do lots of mix and match to find what you want.


ClearCase Commands : File and Version Information
20. To change the comment for a revision.
Many revision control tools won't let you change a comment once committed, on grounds of theology (Tichy didn't permit it), or control freakieness (typically MIS installations).
Cleartool is realistic, and accomodating, so you have a chance to correct and revise comments as needed.
chevent -c "comment" -rep file.name@@/main/version#
will replace an existing comment for revision n on the main trunk.
chevent -c "comment" -rep *
will replace the existing comments on the latest revisions of all files in the current directory.
________________________________________

21. To change the content description for a file.
When you set up a file initially on atria with -mkelem, you can use the -c "comment" qualifier to record a description of the contents of the file.
To change this subsequently:
chevent -c "content description" -rep file.ext@@/main/version#
________________________________________

22. To list the history (versions with comments) of a file.
cl lshist -all -since 08:00 file.ext
will list all changes made to the file since the time specified.
________________________________________

23. To Find out what files have been worked on since a given date.
It is sometimes very useful to get a list of all the changes made to the source tree since a given date.
cl lshist -all -since 31-Jul-96.23:59

________________________________________

24. To find out the file_type of a file.
This is useful to confirm that a file is stored by Cleartool in an appropriate format.
Element type : Use with
text_file : Flat ascii text file
file : Any binary file that you don't mind Cleartool compressing for storage.
compressed_file : Binary file that's already compressed and you want Cleartool to store as-is.
To find this out enter:
describe file.ext
or
describe *
________________________________________

25. To setup cleartool to send email on a file change.
You may want to cause e-mail to be sent to email handle 'supercon' when a particular file has been updated.
This is particularly useful, for example, if you have patched the source in a commercial library and you want to be sure that when the library is upgraded your changes do not get overwritten, as the newest version of the library may or may not have corrected the error your patches peviously corrected.
The e-mail notification acts as a helpful reminder to re-test for the bug which was encountered in the earlier version, and re-update and re-build the commercial library if the library vendor did not fix the problem.
How to do it:
#1: Get VOB access.
#2: Create an e-mail notification script in unix format and add it to atria. For example:

Courtesy aric kollmyer:#!/bin/sh
mailx -s "Cleartool warning: Reminder on zLib update" supercon
ENDMAIL New Version of dialog.cpp
create: $CLEARCASE_PN by:
$CLEARCASE_USER Comment string:
-------------
Someone has really done it now, they installed a new version of zLib!.
This may cause problems as I made the following patches to :
dialog.cpp in the zDialogFrame::create() method to correct .....
So, you had better re-test with the new zApp libraries linked in and
verify that trhe new library fixes this problem
-Me Self (555) 123-4567 Quincy, CA meself@hotmail.com
ENDMAIL
#3: Register the script file as a trigger.
cl mktrtype -nc -element -postop checkout -exec

Example:

cl mktrtype -nc -element -postop checkout -exec
/gl/vobs/netman/src/zapp/source/glenayre.sh glenTrigger
#4: Attach the trigger to an existing element:

cl mktrigger -nattach
Example:

cl mktrigger -nattach glenTrigger dialog.cpp
The examples shown will send e-mail document to supercon when dialog.cpp is checked-out.


ClearCase : Directory Operations

26. To add a new directory to the source tree.
Again a multi part operation as, to achieve transparency, directories are themselves vob (versioned) objects.
#1 First checkout the directory under which you want to create the new sub-directory.
cl co -c "Add directory xxx" .
The period indicates that you are checking out a directory.
#2 mkdir xxx
#3 Give the new directory privilege level 775
cl protect -chmod 775 .
#4 To verify the privilege level is 775
ls -adl .
The directory attribute string should be: d rwr rwr r-x (7 7 5)
These last two steps will ensure that someone else can later manipulate the directory without having to come back to you to beg for the privilege!
________________________________________

27. To rename a directory in the /gl/vobs/netman/src tree.
Eg: /src/ui/thunk_ar
#1 Change to the directory you want to rename eg: /ui/thunk_ar and check it out:
cl co .
#2 Checkout the directory above the directory you want to rename : eg /ui
cl cd..
cl co .
#3 Rename the directory by mv'ing it:
cl mv thunk_ar thunk_31
#4 Check in the directories:
cl ci .
cl cd thunk_31
cl ci .

________________________________________

28. To rename a vob directory or element.
Checkout the directory that contains the directory or element to be renamed.
Do
cl mv oldname newname
for the directory or element. Check it in.


ClearCase Commands : File Add Move Rename
29. To Add a new file to an existing directory.
This is multi part operation as to achieve transparency, cleartool directories are themselves versioned. So ...
#1 First, you have to checkout the directory, because adding a file will change the directory. Enter a checkout comment which can also serve as the check in comment later.
cl co -c "Add file xxx" .
The period indicates that you are checking out a directory.
#2 Make an (empty) file 'element'
cl mkelem -c "file description" file.ext
NOTE! This assumes that the new element has an extension that is known to Cleartool. If not, and you get a gobbledegook message 'Cant pick element type from...' then you must specify an element type eg:
cl mkelem -c "Config stuff" -eltype text_file cb.cfg
The file types are:

text_file : Flat ascii text file
file : Any binary file that you don't mind Cleartool compressing for storage.
compressed_file : Binary file that's already compressed and you want Cleartool to store as-is.
#3 Copy the source code to the new file.
#4 Checkin the file. This will be the first revision.
cl ci -c "comment" file.ext
#5 Check the directory back in.
cl ci .

The default comment you supplied in #1 above sbould be ok.
#6 You should check that you have set up the file with an apppropriate storage type. See: Tip 24 'To find out the file type of a File'.
________________________________________

30. To use symbolic links avoid duplicating an archive file in different directories.
Suppose we have directory A with archived file.ext eg: /../VOBS/.../A/file.ext and we want to have the same file in Directory B.
Rather than duplicate the file, we can place a link in directory /../VOBS/.../B to point to the 'real' archive file in directory A.
To setup the link:

Change to the B directory

Checkout the B directory: cl co -c "Link to file A" .

Change to the A directory

Enter: cl ln -slink /../vobs/.../A/file.ext /../vobs/.../B

Change to the B directory and do a ls -al to inspect the links.

________________________________________

31. To move an atria file from one directory to another.
Check out the source directory :
co .
Check out the target directory :
co .
In the source directory type :
mv file.ext /gl/vobs/netman/src/etc/etc/newfile.ext
________________________________________

32. To make a view private copy of a file.
co -unreserved
________________________________________

33. To Cleartool 'rm' an atria archive file.
First enter the ClearCase tool:
cl
Once INSIDE the ClearCase tool
rm filename
This will remove the archive from the view, but it is still accessible and can be viewed or retrieved if you need it again at some later time.
Warning! If you use rm filename outside Cleartool ie at the shell level the file is GONE!
________________________________________

34. To look at a file that has been Cleartool 'rm'd'.
more .@@/main/5/file.ext@@/main/LATEST
Replace '5' with the last version of the directory that contained the file.
Instead of more you can use less - see man less for info.


ClearCase Commands : Branches

35. To list existing branches.
cl lstype -brtype
________________________________________

36. To Make a branch.
#1 cl mkbrtype Eagle
and enter appropriate comments (doesn't matter where in the tree you are).
#2 mklbtype Eaglebase and enter appropriate comments.
#3 Be in the top level (/src) directory.
mklabel -recurse Eaglebase .
to apply the label throughout the tree.
Wait a long time.
#4 Edit your configspec to use the branch.
Note that if you use the graphical (xlsvtree) viewer to look at a file at this point, the new branch will not show up. No, you didn't do anything wrong. The new branch will not be Created until you check the file out.
________________________________________

37. To merge a branch back to the mainline.
Finally, a chance to screw up bigtime!
#1 Use 'catcs' (Tip 23 : Look at your configspec) to ensure that your view is currently set to the main line.
#2 Change the directory to the root of the source tree.
#3 Label the entire source tree PRE_WHATEVER_MERGE so that if you screw up big time it'll be easier for people to check out files bsu.
#4 cl findmerge . -fver /main/BRANCH_NAME -merge -xmerge
This command will, for all the files on the branch:

present side-by-side Mainline-Branch-FinalMerged windows for all non-trivial merges back to the mainline.

Check out the file, and put the contents of the FinalMerged window in it.

Create a matching file.contrib file.
We're not going to try and verify the merge here and now. Click 'y' to accept the generated merge files, until all the files are processed.
#5 In the root directory you will find a FILExxxx log, listing all the files merged. Print it.
#6 Using the log printout as a control:

Use xlsvtree or something equivalent to diff the checked out file against the predecessor file on the main-line.

Verify the merged file is the way you want it and edit it if not.

Build the app to verify the changes are (syntactically) valid.

Regression testing the app would be nice.

Checkin the verified files to the mainline.

- rm the file.contrib files. Try not to rm the file object by mistake.
#7 Note! There is no such thing as a valid merged .dlg file!
Bring up the latest mainline and latest branch .dlg files in a resource editor and hand tool the changes!


ClearCase Commands : Releases
38. To build a release.
#1 Use catcs / edcs to set up your configspec with
element * LabelToBuildFrom
#2 IMPORTANT! Make sure that you have commented out with the '#' symbol:
#element * /main/LATEST
Failure to do this will mean that later file versions updated AFTER the labelled versions will get into the build!
________________________________________

39. To obtain a directory listing of the source tree for a release.
#1 Use Tip 13 command to get a list of existing labels.
Write down the label for the release you want eg 'RELEASE_2.02'.
Get it right!
#2 Use Tip 43 to add this line to your configspec:
element * RELEASE_2.02
before the existing:
element /gl/vobs/netman/src/docs/.../main/LATEST
element * main/LATEST
Save the configuration, exit, and reply yes to confirm the configspec change.
#3 Open /install/bmp/splash.bmp or title.bmp in PaintBrush to confirm that the configspec change has worked and you are accessing the source version you require.
#4 Do a:
dir s: /s > release.xxx
to do a recursive directory listing of the s: source drive using the view set by the new configspec, and pipe the result to a file.
#5 Don't forget to change your configspec back to normal!
________________________________________

40. To obtain a list of file changes since the last release.
This is useful when you want to do a code review prior to a new release.
In the directory at the top of the src tree:
find . -version 'version(/main/LATEST)
&& !lbtype(RELEASE_3.0)' -exec 'echo $CLEARCASE_PN >> pooh.bar'
This will recursively examine the source tree and output to pooh.bar a list of the files with head versions with no label ie the changes since the last label was applied to all the files.


ClearCase Commands : Administrator

41. To remove (destroy) the latest version(s) of a file.
Warning! Don't do this without being sure you know what you are doing!
This command will chop off the latest version from a file, including versions above it if the version is not the latest one!
rmver file.exe@@/main/version#
________________________________________

42. To look at your configspec.
catcs
This will display the directives that Cleartool is currently using to filter files into your logical 's:' drive. eg:
# Setup to access the main branch
element * /main/LATEST

#Setup to build an entire labelled version
#and ONLY files labelled this way
#element */main/LATEST

must be commented out
#or files added/changed later than this will get included in the build
#element *RELEASE_3.01

# Setup to access the tigger branch
# element * /main/tigger/LATEST
# element * /main/tiggerBase -mkbranch tigger
# element * /main/LATEST -mkbranch tigger
________________________________________

43. To change your configspec.
edcs

Rational Clear Case

Connect to ClearCase
Before using Rational ClearCase commands, you must use one of the following methods to connect to ClearCase:

In the Rational Developer product, click the ClearCase > Connect to Rational ClearCase menu option
In the Rational Developer product, click the Connect to ClearCase icon

Use the ClearCase preference setting Automatically Connect to ClearCase at Startup to establish a ClearCase connection each time you start the Rational Developer product

Connect to Rational ClearCase
To work online with Rational ClearCase from the Rational Developer product, click ClearCase > Connect to Rational ClearCase or click the Connect to Rational ClearCase icon.

You must select this option to make the other ClearCase menu items available. The availability of other ClearCase menu options then varies according to your current selection in the Rational Developer product.

When this option is active, a check mark appears next to the option name and the icon is indented.

To disconnect from ClearCase, click this option again.

You can also set the ClearCase SCM Adapter Automatically connect to ClearCase at startup preference to establish the ClearCase connection.
Automatically connect to ClearCase on startup
This ClearCase SCM Adapter preference establishes a ClearCase connection each time you start a Rational Developer product.

By default, this option is off.





To check out files, edit them, and check them in:

In the Rational Developer product, select the files to check out, right-click, and click ClearCase > Check Out or click the Check Out icon.
In the Check Out Element(s) dialog box, click OK. The file names appear in the Rational Developer product with a checked-out ClearCase decoration next to them.

Edit and save the files in the Rational Developer product.

In the Rational Developer product, select the files to check in, right-click, and click ClearCase > Check In or click the Check In icon.
In the ClearCase Check In Element(s) dialog box, click OK.
By default, ClearCase does not check in files that have not changed. To change this behavior, click Advanced and select the Checkin even if identical option in the ClearCase - Checkin dialog box.

Rational Clear Case Command Reference

How to become a Master in Modify Stage

Modify Stage Purpose
With a Modify stage you can perform five classes of function, but one major limitation is that Modify stage functions are limited to a single argument - straight away arithmetic and concatenation are ruled out, for example.
The five classes of function are:

•null handling
•date/time manipulation
•date/time interval calculation (limited)
•string trimming and substring
•data type conversion
Th Robinson had another great comment pointing out it is better than the Transformer for null handling:

I used to routinely get bitten by the transformer stage rejecting rows in which some fields were participating in derivations or stage variables and the dang things had NULLS in them.
2. Solving the NULLS problem with IF IsNull() for every single field being evaluated in some way can get overly complex and very messy.
Instead I put a Modify stage before the Transformer, call the stage MhandleNull and handle_null() for all fields being evaluated in the transformer. This simplifies the already cumbersome IF THEN ELSE syntax of the transformer and/or the stage variables.

Modify versus Transformer

I've put together a table that compares the Modify Stage to the Transformer Stage to show the limitations - the Modify Stage is a highly specialist stage while Transformer is an all-rounder:

There are a lot of NOs in the Modify column! It is a specialized stage, it is very fast at converting metadata and unlike the Transformer stage it can use job parameters for just about anything. The last No is very important.

There are many wrong paths to Modify Stage enlightenment and very few beacons to light the way, take a torch

So I am learning how to use the Modify Stage through a test harness - I've built some independent DataStage jobs that can be used to test out Modify Stage syntax without having to recompile or change the job. The hardest part of debugging Modify Stage code is that you don't know which specification had the error and it is time consuming to keep changing the code, compiling and retesting. The test harness lets you isolate and test specifications one at a time.

Here are the three jobs.

Using Job Parameters

A Job Parameter is to DataStage what a Swiss Army Knife is to the camper

The examples shown below use job parameters to test different Modify Stage scenarios, there are three ways to set the value of job parameters:

- In the job properties before a compile: this is where you set the default values that come up when the job is run.

- In the Director using Set Job Parameters: this is a convenient place to change the default values created at design time as DataStage will remember and use these defaults between job runs.

- In the Director when a job is run: the set job properties window pops up when you run a job letting you change values, but these values are reset to default the next time you run the job so it's not a good place to put complex Modify test code while you are debugging.

Courtesy: Vincent McBurney
-----------------------------------------------------------------------------------
Learning the Modify Stage

The Modify stage is a metadata panel beater. It does just one thing: converts columns from one data type to another. It is quite restrictive, you cannot nest functions or use constants as values. Almost all the functions in the Modify stage are also available in the all rounder Transformer stage.
The Transformer is an automatic, the Modify Stage is a manual

Transformers are easy to use, which is one of the reasons why DataStage has been successful as it is the most commonly used stage. When adding derivation / mapping / transformation code into the text entry windows there is a right mouse click menu for finding a list of transformation functions:

It all started a couple of AscentialWorld conferences ago when Ascential DataStage experts started telling people to avoid the parallel transformer if they wanted fast performance in DataStage jobs. If a parallel job is a racing car the transformer stage was a roof rack and luggage.

Historically parallel jobs did not even have a transformer stage.

Back when the parallel engine was a product called Orchestrate there was a Transform function that worked much the same way as other parallel commands. It required a lot of manual coding. When Ascential turned Orchestrate into parallel jobs they took the transformer stage from Server Edition and sat it over the top of the parallel transform function. This is when the trouble began. The transformer stage and the Transform function didn't quite fit together, so each parallel transformer needs a C++ compilation, a kind of automated custom stage build, even in the latest versions. The message is that made it slower then other parallel stages.

The Allrounder Stage
Most of the functions of a transformer can be performed by other stages.
Transformer constraints = filter stage
Transformer metadata conversion = Modify stage
Transformer copy of fields and multiple outputs = Copy stage
Transformer creation of new fields = Column Generator stage
Transformer counters = Surrogate Key Stage

As you can see each of the parallel stages are highly specialised while the transformer is an allrounder. The specialised stages are faster because they do not carry as much overhead.

Ease of Use versus Performance
Let's start with the Modify stage. The only people out there who enjoy using this stage are also experts at crytic crosswords or were part of the team that cracked the enigma code in world war two or are members of opus dei and are looking for the DataStage equivelent of self flagulation. With no friendly right mouse click menus, no red text highlighting syntax errors, not warnings or helpful messages on compile there is simply no easy way to ensure you have correct Modify stage code without compiling and waiting for a (hopefully) helpful error message. Trial and error is the order of the day.

The Filter stage is not much better.

The Column Generator stage is just bizarre and involves navigating your way into the column properties like a trip through the pipe system on Prison Break.

The Transformer is hands down the best stage to write any type of transformation code in the entire DataStage product given the right mouse click menu, the red text marking for syntax errors and the grid layout.

Robustness
The transformer stage is one of the few stages that has a robust reject link for rows that fail transformation. The Modify and Filter stages lack this link so if a piece of data defies logic the job will abort rather then drop a row down a reject link.