Sqlldr - getting all nulls in target table

Question

Database: 10gR2 on WinXP
I am trying to load about 4000 of 7.5 million lines into the database from a 390MB file (there are 1522 of these files in total).
Goal:
1. When the line starts with "ScanHeader" or "position = ", just insert the line into TEST1.
2. When the line starts with "packet #", break it up into columns and insert into TEST2.
In fact, I only want three values out of the line. For example, for the line:
"Packet # 2, intensity = 0.000000, mass/position = 50.250000"
I want "2", "0.000000", and "50.250000" to be inserted into a table (and only if the intensity (the second value) is greater than 25).
(Note: I was thinking I would just load them all and then delete the ones I don't want, but if there is a way to exclude them up front, I would love to see it).
The loading into TEST1 seems to be working, but my first several stabs at loading into TEST2 results in the correct number of rows with source_file and seq set properly, but NULL for all the other columns. I suspect this is probably an easy one, but the correct search terms are eluding me.
My rust-encrusted loader skills were somewhat limited in the first place, so any suggestions beyond fixing the main issue are appreciated.
Thanks!
-Tom
CONTROL FILE:
OPTIONS(DIRECT=TRUE, ROWS=100000)
UNRECOVERABLE
LOAD DATA
INFILE '081017102.txt'
BADFILE '081017102.bad'
TRUNCATE
INTO TABLE TEST1
WHEN (1:10) = 'ScanHeader'
(source_file constant '011017102.txt',
content position(1:200) char,
SEQ recnum)
INTO TABLE TEST1
WHEN (1:10) = 'position ='
(source_file constant '011017102.txt',
content position(1:200) char,
SEQ recnum)
INTO TABLE TEST2
WHEN (1:8) = 'Packet #'
fields terminated by ' '
trailing nullcols
(source_file constant '011017102.txt',
seq recnum,
fpacket,
fpoundsign,
packet_number,
fintensity,
fequals1,
fcontent,
fmplabel,
fequals2,
mp
SAMPLE DATA FROM ONE FILE (partial file):
RunHeaderInfo
dataset_id = 0, instrument_id = 0
first_scan = 1, last_scan = 318, start_time = 0.002282, end_time = 0.997597
low_mass = 50.000000, high_mass = 1000.000000, max_integ_intensity = 41286.089844, sample_volume = 0.000000
sample_amount = 0.000000, injected_volume = 0.000000, vial = 0, inlet = 0
err_flags = 0, sw_rev = 1
Operator =
Acq Date =
comment1 =
comment2 =
acqui_file =
inst_desc =
sample volume units =
sample amount units =
Injected volume units =
Packet Position = 38630
Spectrum Position = 14603130
ScanHeader # 1
position = 1, start_mass= 50.000000, end_mass = 1000.000000
start_time = 0.002282, end_time = 0.000000, packet_type = 0
num_readings = 11400, integ_intens = 15376.646484, data packet pos = 0
uScanCount = 0, PeakIntensity = 1098.594238, PeakMass = 75.333328
Scan Segment = 0, Scan Event = 0
Precursor Mass
Collision Energy
Isolation width
Polarity negative, Profile Data, Full Scan Type, MS Scan
SourceFragmentation Off, Type Ramp, Values = 0, Mass Ranges = 0
Turbo Scan Off, IonizationMode Electrospray, Corona Any
Detector Any, Value = 0.00, ScanTypeIndex = -1
DataPeaks
Packet # 0, intensity = 0.000000, mass/position = 50.083333
saturated = 0, fragmented = 0, merged = 0
Packet # 1, intensity = 0.000000, mass/position = 50.166667
saturated = 0, fragmented = 0, merged = 0
Packet # 2, intensity = 0.000000, mass/position = 50.250000
saturated = 0, fragmented = 0, merged = 0
(...11,900 more packets for each of 220 ScanHeaders...)
LOG FILE FROM LOADING A PARTIAL FILE:
SQL*Loader: Release 10.2.0.1.0 - Production on Tue Oct 21 11:24:11 2008
Copyright (c) 1982, 2005, Oracle. All rights reserved.
Control File: pos.ctl
Data File: 081017102.txt
Bad File: 081017102.bad
Discard File: none specified
(Allow all discards)
Number to load: ALL
Number to skip: 0
Errors allowed: 50
Continuation: none specified
Path used: Direct
Silent options: FEEDBACK and DISCARDS
Load is UNRECOVERABLE; invalidation redo is produced.
Table TEST1, loaded when 1:10 = 0X5363616e486561646572(character 'ScanHeader')
Insert option in effect for this table: TRUNCATE
Column Name Position Len Term Encl Datatype
SOURCE_FILE CONSTANT
Value is '011017102.txt'
CONTENT 1:200 200 CHARACTER
SEQ RECNUM
Table TEST2, loaded when 1:8 = 0X5061636b65742023(character 'Packet #')
Insert option in effect for this table: TRUNCATE
TRAILING NULLCOLS option in effect
Column Name Position Len Term Encl Datatype
SOURCE_FILE CONSTANT
Value is '011017102.txt'
SEQ RECNUM
FPACKET NEXT * WHT CHARACTER
FPOUNDSIGN NEXT * WHT CHARACTER
PACKET_NUMBER NEXT * WHT CHARACTER
FINTENSITY NEXT * WHT CHARACTER
FEQUALS1 NEXT * WHT CHARACTER
FCONTENT NEXT * WHT CHARACTER
FMPLABEL NEXT * WHT CHARACTER
FEQUALS2 NEXT * WHT CHARACTER
MP NEXT * WHT CHARACTER
Table TEST1, loaded when 1:10 = 0X706f736974696f6e203d(character 'position =')
Insert option in effect for this table: TRUNCATE
Column Name Position Len Term Encl Datatype
SOURCE_FILE CONSTANT
Value is '011017102.txt'
CONTENT 1:200 200 CHARACTER
SEQ RECNUM
Table TEST1:
1 Row successfully loaded.
0 Rows not loaded due to data errors.
34234 Rows not loaded because all WHEN clauses were failed.
0 Rows not loaded because all fields were null.
Table TEST2:
11400 Rows successfully loaded.
0 Rows not loaded due to data errors.
22835 Rows not loaded because all WHEN clauses were failed.
0 Rows not loaded because all fields were null.
Table TEST1:
1 Row successfully loaded.
0 Rows not loaded due to data errors.
34234 Rows not loaded because all WHEN clauses were failed.
0 Rows not loaded because all fields were null.
Bind array size not used in direct path.
Column array rows : 5000
Stream buffer bytes: 256000
Read buffer bytes:20971520
Total logical records skipped: 0
Total logical records read: 34235
Total logical records rejected: 0
Total logical records discarded: 22833
Total stream buffers loaded by SQL*Loader main thread: 5
Total stream buffers loaded by SQL*Loader load thread: 0
Run began on Tue Oct 21 11:24:11 2008
Run ended on Tue Oct 21 11:24:12 2008
Elapsed time was: 00:00:01.09
CPU time was: 00:00:00.10

Answer 1

I don't think that is the issue, but thanks for the suggestion.
I eliminated the directives to load data into TEST1 from the control file and then, surprise surprise, the load into TEST2 started working. I added the TEST1 directives back in after the TEST2 directive and all appears to be working now. I don't understand exactly why, but at this point I will accept the result.
I have changed the control file to what you see below. For the TEST2 load, any suggestions on how to skip the records where the intensity is less than 25?
REVISED CONTROL FILE:
OPTIONS(DIRECT=TRUE, ROWS=100000)
UNRECOVERABLE
LOAD DATA
INFILE '081017102-partial.txt'
BADFILE '081017102.bad'
TRUNCATE
INTO TABLE TEST2
WHEN (1:8) = 'Packet #'
fields terminated by ' '
trailing nullcols
(source_file constant '011017102',
seq recnum,
fpacket filler,
fpoundsign filler,
packet_number "replace(:packet_number, ',', '')",
fintensity filler,
fequals1 filler,
intensity "replace(:intensity, ',', '')",
fmplabel filler,
fequals2 filler,
mass_position
INTO TABLE TEST1
WHEN (1:10) = 'ScanHeader'
(source_file constant '011017102',
content position(1:200) char,
SEQ recnum)
INTO TABLE TEST1
WHEN (1:10) = 'position ='
(source_file constant '011017102',
content position(1:200) char,
SEQ recnum)