Special Characters in Incoming Request in VSM

[Durai Kannan]

I created VSI using IBM MQ Native protocol and sending the original request to listener with some tags have special characters. When i checked lisa.vse.request, its having latin characters instead of that special characters. I need a value same as original request. Could you please help me on this.

 

Original Request:

<?xml version="1.0" encoding="UTF-8">
<request>
<tag1>test tag 2.5</tag1>
<tag2>ISO 0 – ½ test<tag2>
</request>

 

lisa.vse.request:

<?xml version="1.0" encoding="UTF-8"?>
<request>
<tag1>testátagá2.5</tag1>
<tag2>ISO 0 û ¢ test<tag2>
</request>

[Kevin.Bowman]

What Data Protocols are you using?

What character set (CCSID) is your IBM MQ request message using?

How was the VS created?  Recording or RR pairs? 

Can you open your VSI in workstation, navigate to one of the requests, open the Attributes tab, and copy/paste the contents of the "recorded_raw_request_bytes" attribute?

[Durai Kannan]

Hi Kevin,

 

Please find my comments below.

 

1) XML Data protocol
2) encoding="UTF-8"
3) RR pairs
4)

<request>
<tag1>ISO 0 � � EURO</tag1>
<tag2>A�B�C</tag2>
</request>

[Kevin.Bowman]

The "recorded_raw_request_bytes" attribute should be in base64.  Just copy/pasting your request in here isn't going to let me see the actual bytes because that goes through who knows how many layers of text encodings/decodings.

 

If you're using RR pairs then can you attach a few examples?  Do not copy/paste, just attach the files or zipped up files.

 

Also, 'encoding="UTF-8"' is not a character set.  I'm talking about the Coded Character Set ID, or CCSID, which is part of every IBM MQ message's header.

[Durai Kannan]

Hi Kevin,

 

Please find the my comments below,

 

1) CCSID = 1208

2) recorded_raw_request_bytes=PHJlcXVlc3Q DQo8dGFnMT5JU08gMCCWIL0gRVVSTzwvdGFnMT4NCjx0YWcyPk9SR6BDQVJST1SgQkFHPC90YWcyPg0KPHRhZzM UGF26SBEaWFtYW506TwvdGFnMz4NCjwvcmVxdWVzdD4

3) PFA RR pairs for the same

Attachment(s)

RR.zip   409 B 1 version

[Kevin.Bowman]

CCSID 1208 is UTF-8, which also happens to be the default encoding when importing RR pairs. 

 

However, your text data is not a valid UTF-8 encoding.  It looks like extended ASCII, otherwise known as Windows-1252.  That would be CCSID 1252.

 

You need to specify Windows-1252 as the character set when importing your RR pairs:

 

You will also need to either fix your application to send the correct CCSID, 1252, or enter an override CCSID when configuring the IBM MQ Native transport protocol.  Make sure the 'PRO' button is enabled, click the gear icon next to the Proxy Request Queue, and fill in' Override Character Set' with "1252":

 

If your application is expecting the response to be Windows-1252 encoded, then you will probably also have to make sure your response message has the right CCSID.  I believe you can define a response-side sidecar file with your RR pairs containing the property

msg.characterset=1252

 

Or you can edit the VSI manually to set the msg.characterset meta-data property:

[Durai Kannan]

Thanks Kevin for detailed explanations.

 

Could you please answer the below queries,

 

1) What is the purpose of doing override character set (1252 for our case). Will it change character as 1252 if we send request format as any other except 1252?

 

2) if we change msg.characterset=1252, then is it also mandatory to set msg.encoding as well?

[Kevin.Bowman]

> 1) What is the purpose of doing override character set (1252 for our case).

You need to set the override character set because, in the messages from your application, the character set defined in the message headers conflicts with the character set used to actually encode the message body. It forces DevTest to use Windows-1252 to decode the message body regardless of what the message's character set header says.

 

Alternative solutions would be to either change your application to send 1252 in the character set header, or change you application to use UTF-8 when encoding the message body.

 

> Will it change character as 1252 if we send request format as any other except 1252?

I'm not sure I understand the question.  If you send a different value in the character set header while still encoding the message body the same way then it makes no difference; override character set overrides whatever the message header's character set is.  If you actually encode the body with a different character set then you will either have to change the override character set in DevTest or, if it now matches the message header character set, remove the override character set from DevTest.

 

> 2) if we change msg.characterset=1252, then is it also mandatory to set msg.encoding as well?

msg.encoding controls how numeric header values are encoded.  It has nothing to do with the message body character encoding.

[Durai Kannan]

Thanks Kevin for the clarifications.