Accepting Attachments into OSB, forwarding via JMS

Scenario

We needed to provide an OSB service (hrs.wfm.cvs.PI.Worker_1.0) to accept a SOAP request plus attached XML file containing a set of records. We wanted to discard the request, but forward the attached file content onto a JMS queue for further processing. A BPEL would de-batch these records and insert the records into a destination database, effectively an ETL process.

The platform was SOA and OSB 11.1.1.7. These are the steps we used.

Approach

1. Define Multi-part WSDL with attachments

It is straightforward to define a WSDL that accepts attachments.

(1.1) Start with a standard WSDL, defined with request and response messages as you normally would.

(1.2) Add the {http://schemas.xmlsoap.org/wsdl/mime/} namespace to the top of your WSDL

e.g.: in hrs.wfm.cvs.PI.Worker_1.0.wsdl

<wsdl:definitions  
name="hrs.wfm.cvs.PI.Worker"  
xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/"  
xmlns:xsd="http://www.w3.org/2001/XMLSchema"  
xmlns:flt="http://rxr.example.com/ebo/util.flt.Fault"  
xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"  
xmlns:tns="http://rxr.example.com/cvs.PI/hrs.wfm.Worker"  
targetNamespace="http://rxr.example.com/cvs.PI/hrs.wfm.Worker"  
xmlns:mime="http://schemas.xmlsoap.org/wsdl/mime/">

... rest of WSDL....  

</wsdl:definitions>  

(1.3) Add your attachment-part to your request message type, e.g. the part named workerDataClob below.

e.g.: in hrs.wfm.cvs.PI.Worker_1.0.wsdl

<wsdl:message name="uploadWorkerData">  
    <wsdl:part name="payload" element="tns:uploadWorkerData"/>  
    <wsdl:part name="workerDataClob" element="tns:workerDataClob"/>  
</wsdl:message>  
<wsdl:message name="uploadWorkerDataResponse">  
    <wsdl:part name="payload" element="tns:uploadWorkerDataResponse"/>  
</wsdl:message>  

If you're uploading text/xml then this element can be typed as a user-defined business object type.

e.g.: in hrs.wfm.cvs.PI.Worker_1.0.xsd

<xsd:element name="workerDataClob" type="pi:ReplicateEmployeeMessage_Async_V01"/>  

If you're uploading binaries, like PDFs or Word Docs, you would type your attachment element as base64Binary

e.g.: in hrs.wfm.cvs.PI.Worker_1.0.xsd

<xsd:element name="workerDataClob" type="xsd:base64Binary"/>  <!-- if uploading binaries -->  

(1.4) In your WSDL binding, denote the operation as being multi-part. Note how the operation input section is defined.

e.g.: in hrs.wfm.cvs.PI.Worker_1.0.wsdl

<wsdl:binding name="hrs.wfm.cvs.PI.Worker.Binding" type="tns:hrs.wfm.cvs.PI.Worker">  
    <soap:binding style="document" transport="http://schemas.xmlsoap.org/soap/http"/>  
    <wsdl:operation name="uploadWorkerData">  
        <soap:operation soapAction="uploadWorkerData" style="document"/>  
        <wsdl:input>  
              <mime:multipartRelated>  
                  <mime:part>  
                      <soap:body parts="payload" use="literal"/>  
                  </mime:part>  
                  <mime:part>  
                      <mime:content part="workerDataClob" type="text/xml"/>  
                  </mime:part>  
              </mime:multipartRelated>  
        </wsdl:input>  
        <wsdl:output>  
            <soap:body use="literal"/>  
        </wsdl:output>  
        <wsdl:fault name="svcFault">  
            <soap:fault name="svcFault" use="literal"/>  
        </wsdl:fault>  
    </wsdl:operation>  
</wsdl:binding>  

(1.5) It's helpful to validate your WSDL and XSD in something like XmlSpy at this point.

2. Prepare Sample SoapUI Request

MIME attachments don't behave as you might expect from the OSB test console, so it's best to prepare a SoapUI test project.

(2.1) From SoapUI, generate a project based on your WSDL.

(2.2) Generate a sample data file which you will attach to your SOAP request. This can be stored in the same folder as the project.

(2.3) Define an attachment for the SOAP request and allocate your file to the correct request part from the drop down list.

(2.4) You should also set the ContentID of the request Part with a known name, preferably the same name as the part itself. This lets us query the specific part from OSB.

e.g.: in the SOAP request below, an attachment (highlighted in yellow) has been added representing the workerDataClob request-part which the WSDL defined in step 1.3 above. We've also chosen to designate the part's content id with the same name.

(2.5) Confirm that your attachment file content is being sent

When you fire the SoapUI request, check the raw request view to make sure that your file content has been submitted in the MIME part.

If you see only the filename, but not the content, then the content itself has not gone over to OSB. Try validating your soapui request to see if your attachment filepath is still valid.

Notice also that SoapUI has wrapped the workerDataClob content-id with <brackets>

e.g.: SoapUI Raw Request view after firing your request

POST http://osb-server:8003/evs/hrs.wfm.cvs.PI.Worker_1.0 HTTP/1.1  
Accept-Encoding: gzip,deflate  
Content-Type: multipart/related; type="text/xml"; start="<rootpart@soapui.org>"; boundary="----=_Part_8_2535272.1459905255478"  
SOAPAction: "uploadWorkerData"  

------=_Part_8_2535272.1459905255478  
Content-Type: text/xml; charset=UTF-8  
Content-Transfer-Encoding: 8bit  
Content-ID: <rootpart@soapui.org>  

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:hrs="http://rxr.example.com/cvs.PI/hrs.wfm.Worker" xmlns:util="http://rxr.example.com/ebo/util.cmn.EBMHeader">  
   <soapenv:Header/>  
   <soapenv:Body>  
    ..etc...  
   </soapenv:Body>  
</soapenv:Envelope>


------=_Part_8_2535272.1459905255478  
Content-Type: text/xml; charset=us-ascii; name=sample-2-PI.xml  
Content-Transfer-Encoding: 7bit  
Content-ID: <workerDataClob>  
Content-Disposition: attachment; name="sample-2-PI.xml"; filename="sample-2-PI.xml"  

....The file content should be visible here....

If your content appears to be cropped in the raw request view then check your SoapUI settings under

> File > Preferences > UI Settings > Size of raw request message to show

3. Configure Paging/Buffering on the OSB Proxy

Our use case was to accept a potentially large textual SOAP attachment and pass through to JMS.

(3.1) Develop and configure your OSB Proxy as you normally would

(3.2) Enable Paging from the Message Handling Tab.

Use buffering and paging so we don't load a Godzillagram into OSB memory, but spool it to/from disk instead.

(3.2a) Consider securing your on-disk buffer location

If security is a consideration while you buffer content to disk then you can designate a secured folder to act as your temporary folder for paging.

(3.3) Define a (JMS) Messaging Service *.biz file of request-type Text (not request-type XML). This simplifies the OSB dispatch logic

(3.4) In the OSB Pipeline, assign the attached content to $body in order to Publish to our *.biz service.

The content from our SoapUI request is available via OSB's built-in $attachments variable.

Specifically we're interested in the section identified by ContentID from step 2.4 above.

$attachments/ctx:attachment[ctx:Content-ID='<workerDataClob>']/ctx:body/*[1] 

When replacing the content of $body, replace the 'node contents' of $body, not the 'entire node'. The content is copied by reference, not copied by value.

(3.5) nullify the $attachments variable, otherwise it causes OSB to publish a MIME package to JMS instead of our raw file content

e.g.: in OSB, before we invoke our Publish action

Assign [ () ] to [ attachments ]  

This doesn't nullify your actual content, it just nullifies the reference from the $attachments variable.

At this point, you should be able to monitor the JMS queue in WLS for the arrival of your payload.

4. Consume Content from BPEL

Our scenario was to accept a set of 4000+ records in XML format and de-batch them for insertion into a destination database.

Oracle JMS JCA Adapter provides support to stream payload. When you enable this feature, the payload is streamed to a database instead of getting manipulated in the SOA run time as in a memory DOM.

However, when trying to assign elements from a streamed input to a local variable I was seeing an exception:

java.lang.ClassCastException - java.lang.String cannot be cast to org.w3c.dom.Element

It's possible to stream the result of an xslt to disk (section 45.1.3.4).

There is also a doStreamingTranslate() in BPEL, but it was not clear to me from the documentation how this should be used.

Each of these options appear to be intended for translating an attachment in transit and sending back to disk, rather than pulling a window of records into memory for de-batching.

As a result we end up taking the whole input into BPEL memory. If large payloads become too onerous for SOA then one possibility is to ask the sender to send multiple smaller files for processing, perhaps applying an OWSM max_request_size_policy on the receiving Proxy.

Potential Issues

As tools go, a designated ETL tool may be a more natural fit for this scenario, customer constraints allowing. Potential issues when using SOA and OSB include...

  1. FTP would be a more reliable transport than Http for uploading large files to a service.
  2. If transactionality is enabled then we have long running transactions in middleware which might time-out
  3. Records must all be read into middleware memory at some point, risking a possible Out Of Memory exception.
  4. Consequently, longer running garbage collection on the Servers may cause the JVMs to stall periodically.
  5. The database will treat the updates as OLTP updates instead of Bulk Load so the database will do much more work to update database indexes, transaction logs etc.

Gotchas

Here are a few points that might side-track you...

1. Message dispatched to JMS as multi-part MIME type

When pushing $attachments/ctx:attachment[ctx:Content-ID='<workerDataClob>']/ctx:body/*[1] onto JMS, it was going over as a MIME package, with metadata and part definitions, not as a raw XML payload.

e.g.: JMS queue content includes a whole MIME package instead of just the raw file content.

MIME-Version: 1.0  
Content-Type: multipart/related; boundary=MIME_Boundary; start="<rootpart@soapui.org>"; type="text/xml"  

--MIME_Boundary  
Content-ID: <rootpart@soapui.org>  
Content-Type: text/xml; charset=UTF-8  
Content-Transfer-Encoding: 8bit  

<hrs:uploadWorkerData version="1.0" xmlns:hrs="http://rxr.example.com/cvs.PI/hrs.wfm.Worker" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:util="http://rxr.example.com/ebo/util.cmn.EBMHeader">  
    <util:header>  
        ...etc...  
    </util:header>  
    <hrs:body>  
        <hrs:originCompany>NSWEC</hrs:originCompany>  
    </hrs:body>  
</hrs:uploadWorkerData>  

--MIME_Boundary  
Content-ID: <workerDataClob>  
Content-Type: text/xml; charset=Cp1252; name=sample-2-PI.xml  
Content-Transfer-Encoding: base64  
Content-Disposition: attachment; name="sample-2-PI.xml"; filename="sample-2-PI.xml"  

...file content here...  

--MIME_Boundary-- 

This is because of OSB Dispatch logic when sending to an XML messaging service (*.biz) while attachments are present:

If the outgoing message type is XML and attachments are defined in the attachments variable, a MIME package is created from the XML message and the attachment data. In the case of a null XML message, the corresponding MIME body part is empty.

You might notice this if your messages are not coming through to BPEL. Check your BPELs 'Faults and Rejected Messages' tab in EM console.

Disable consumption of messages from the JMS servers, send another message, then view the content of your JMS queue from the WLS console.

Make sure your JMS business service in OSB is Text-based rather than XML-based.

2. nullify the $attachments before you publish to JMS

When you assign your workerDataClob content from the $attachments into $body, OSB now publishes a multipart JMS message showing the workerDataClob twice.

MIME-Version: 1.0  
Content-Type: multipart/related; boundary=MIME_Boundary;  
    start="<rootpart@soapui.org>"; type="text/plain"  

--MIME_Boundary  
Content-ID: <rootpart@soapui.org>  
Content-Type: text/plain; charset=UTF-8  
Content-Transfer-Encoding: 8bit  

<?xml version="1.0" encoding="UTF-8"?>  
<ns2:ReplicateEmployee_Async_V01 xmlns:ns2="https://bhpbilliton.net/services/Employee">  
    ...snip...  
</ns2:ReplicateEmployee_Async_V01>  

--MIME_Boundary  
Content-ID: <workerDataClob>  
Content-Type: text/xml; charset=us-ascii; name=sample-2-PI.xml  
Content-Transfer-Encoding: 7bit  
Content-Disposition: attachment; name="sample-2-PI.xml"; filename="sample-2-PI.xml"  

<?xml version="1.0" encoding="UTF-8"?>  
<ns2:ReplicateEmployee_Async_V01 xmlns:ns2="https://bhpbilliton.net/services/Employee">  
    ...snip...  
</ns2:ReplicateEmployee_Async_V01>  

--MIME_Boundary--

This is because of the OSB dispatching logic mentioned above.

Remember to nullify your OSB $attachments variable by assigning the null set, '()', to the $attachments variable before you publish $body to JMS.

3. Debugging your OSB $attachment content

In OSB, streamed Attachments are passed-by-reference rather than passed-by-value, so if you tried to use an OSB Report task to confirm the content of your $body variable, you would see the reference to the attachment object rather than the de-referenced content itself.

<con:binary-content ref="cid:-15d7c80d:153e4d87337:3be6"/>  

This is correct and as expected.

You could possibly use an OSB Log task to write it to the logs, or view the content on the JMS queue itself from WLS console. Turn off message consumption from the queue's JMS Servers to allow the messages to sit for inspection.

Alternatively, turn off streaming while you develop with small payloads.

Garret O'Kelly

Read more posts by this author.