1. 程式人生 > >Spark轉GemFire任務(一)

Spark轉GemFire任務(一)

pin val found reg for ever have class chan

ADMG-40 2.2.2.4 - Global Tax Warehouse (GTW) CoA mapping -- Both BtB and Project 1

Revision 7/15, change all latam to mdg

Create GFS account mapped to GTW using FAGL_011ZC, write to /tax/gfs_gtw_mapping

and go through /btb_mdg/ska1 to check all 10 digit GL account can be mapped, if not, write to exception region /tax/master_data_exception

Use BTB_MDG.SKA1 (if BTB_MDG is not ingested, use BTB_LATAM.SKA1):

  1. For every record of SKA1 (select every record from SKA1 where KTOPL = ‘JNJG’ and XSPEB <> ‘X’ ) -- collection A

collectionA:

Path: /btb_mdg/ska1

Filter: KTOPL = ‘JNJG’ and XSPEB <> ‘X’

Column: saknr

自己本身

  1. select records in BTB_MDG.FAGL_011ZC (where VERSN=’GTW’ AND KTOPL=’JNJG’) – collection B

collectionB:

Path: /btb_mdg/fagl_011zc

Filter: trim(versn) = ‘GTW’ and ktopl = ‘JNJG’

Column: from vonkt

To biskt

自己本身的

  1. match each A.SAKNR with [B. VONKT , B. BISKT], B.VONKT is the starting number, B. BISKT is the ending number, both numbers are included in the range, when a match is found,

Validate wheather each item in collectionA in the range of collectionB

判斷是否 A包含於B

4. Get gfsCode (A.SAKNR) and gtwCode (B.ERGSL) description:

for gfsCode -- /btb_mdg/skat: spras:E AND ktopl: JNJG AND saknr:%A.SAKNR%, let it be D, D.txt50 is the gfsCode description.

gtwCode -- /btb_mdg/fagl_011qt : versn:GTW AND spras:E AND ergsl:%gtwCode%, let it be E, E.txt45 is the gtwCode description

gfsCode:

Select txt50 as gfsCodeDescription from /btb_mdg/skat where spras = ‘E’ and ktopl = ‘JNJG’ and saknr in collectionA.saknr

Asaknr有關系

gtwCode:

Select txt45 as gtwCodeDescription from /btb_mdg/fagl_011qt where trim(versn) = ‘GTW’ and spras = ‘E’ and ergsl in collectionB.ergsl

Bergsl有關系

  1. save it into /tax/gfs_gtw_mapping: <gfs:A.SAKNR, gfs_description:D.txt50, gtw:B.ERGSL, gtw_description:E.txt45>

Insert into /tax/gfs_gtw_mapping(

Gfs : collectionA.saknr,

Gfs_description : txt50,

Gtw : collectionB.ergsl,

Gtw_Description : txt45

)

Validation:

3.1 If there is no match, write exceptionCode:INVALID_GTW_MAPPING, excepitonMessage: No Match on GTW Code for GFS code % A.SAKNR%

items in CollectionA but not in the range of CollectionB will be written into Exception

exceptionCode : INVALID_GTW_MAPPING

exceptionMessage : No Match on GTW Code for GFS code % A.SAKNR%

all [B. VONKT , B. BISKT] should not have overlap, check all [B. VONKT , B. BISKT] do not overlap by sorting B. VONKT and B. BISKT and make sure B.VONKT(n+1) < B. BISKT (n), ), if there is invalid range, exceptionCode: INVALID_GFS_MAPPING, exceptionMessage overlapping range is found, [B(n). valfrom, B(n).valto] and [B(n+1). valfrom, B(n+1).valto],

All the ranges in CollectionB cannot have overlap,if it does,written into Exception

exceptionCode: INVALID_GFS_MAPPING

exceptionMessage overlapping range is found, [B(n). valfrom, B(n).valto] and [B(n+1). valfrom, B(n+1).valto]

6. Saving exception:

Exception should be saved to /tax/master_data_exception

{

sourceSystem: btb_mdg,

exceptionCode:

exceptionMessage:

timestamp

}

Spark轉GemFire任務(一)