When implementing Endevor Web Services, one optional but important configuration step comes into play if your environment involves multiple EBCDIC code pages. This configuration ensures that data is correctly translated between the EBCDIC character sets used in your system and the client-side character set (typically UTF-8) that is expected for processing and display.
If your Endevor system accesses Elements across an inventory that spans multiple EBCDIC code pages, you’ll need to configure Endevor Web Services to handle these differences. This process involves setting up a Code Page Configuration File that maps the correct code pages for various Element locations.
In this blog, we’ll walk through why this is necessary, how to configure the system, and the specific actions impacted by this configuration.
Why Configure Multiple EBCDIC Code Pages?
Endevor Web Services interacts with Endevor Elements stored in an inventory, which consists of multiple environments, systems, and subsystems. These Elements may be encoded using different EBCDIC code pages, depending on the system they originated from. Accurate translation of character encoding is key when accessing these Elements via Web Services, particularly through Zowe CLI, Eclipse Plugin, or Explorer for Endevor, to ensure data integrity and prevent unreadable characters on the client side (often displayed in UTF-8).
1. Endevor Inventory May Contain Elements with Different EBCDIC Code Pages
- Different teams, applications, or legacy systems might use distinct EBCDIC code pages based on regional or historical configurations.
- Some elements may use EBCDIC 037 (U.S./Canada), while others may use EBCDIC 500 (International/European) or other variations.
- If Web Services assumes a single EBCDIC code page, Elements stored in different encodings may be misinterpreted, leading to incorrect translations.
- Endevor has no information about the specific EBCDIC code page used to encode each element, it depends on the EBCDIC code page set up in the user’s 3270 emulator when they edited the element.
2. Web Services Must Ensure Correct Translation to Client-Side Encoding
- Translation Process: Endevor Web Services translates Element content from EBCDIC to UTF-8 before sending it to a client application.
- Risk of Incorrect EBCDIC Code Page: If an incorrect EBCDIC code page is used during translation, characters can become garbled, leading to potential data integrity issues.
- Data Integrity Risk in Round-Trip Process: There is a specific risk that after a round-trip (retrieve + add/update), the incorrectly translated characters may be corrupted in Endevor. This occurs when two or more characters translate to the same character in the target encoding, potentially causing data loss or misrepresentation.
- Example Issue: If an Element stored in EBCDIC 500 is interpreted using EBCDIC 037, certain characters (such as square brackets, curly braces, or special symbols) may be misrepresented. If these characters are then retrieved, modified, and added back, the round-trip process could cause the corrupted characters to be overwritten or lost.
- Mitigation: To ensure data integrity, it’s critical to use the correct EBCDIC code page during translation, preventing character corruption throughout the entire process.
3. Some Users Switch Code Pages in Their Terminal Emulator
- If a user needs to manually change the EBCDIC code page in their terminal emulator to correctly view or edit different Elements, this is a strong indication that multiple code pages exist in the Endevor inventory.
- Web Services needs to handle this variation automatically to prevent incorrect translations.
4. Code Page Configuration File Provides a Solution
- The Code Page Configuration File allows Endevor Web Services to associate different code pages with different Elements based on their location in the inventory.
- Elements in System A could be translated using EBCDIC 037
- Elements in System B could use EBCDIC 500
- Ensuring the correct EBCDIC-to-UTF-8 translation happens dynamically without manual intervention.
Which Actions Are Affected?
The Code Page Configuration File affects the translation of Element content for Element Types using DATA FORMAT = TEXT during the following actions, when invoked via the REST API:
- Retrieve Element: When fetching an Element, the system will check the Code Page Configuration File to ensure proper translation to the client-side charset.
- Add and Update Element: Any time an Element is added or updated, the correct EBCDIC code page is used to translate the content.
- Print Element (Browse, History, or Changes): The printed Element content will be correctly encoded using the appropriate code page during browsing or when reviewing history/changes.
- Submit SCL (Software Control Language): When submitting an SCL request to perform the aforementioned actions, the system will ensure that the correct translation is applied.
Important Considerations
- Avoid utilizing the Submit SCL Generic Endpoint when implementing custom integrations with the Endevor REST API: While it’s possible to use the Submit SCL generic endpoint for these actions, it’s not recommended in environments with multiple EBCDIC code pages. Using this endpoint might result in an incorrect code page being used, especially when wildcarding is involved in the request or when multiple Elements are affected. This could lead to data corruption or misinterpretation of characters.
- Inventory Location and Wildcarding: Ensure that the inventory locations are correctly defined in the configuration file. Wildcarding or ambiguous locations in requests may result in incorrect code page handling.
Integrating the Code Page Configuration with Web Services
Step-by-step instructions on how to update the Web Services configuration file to link it with the Code Page Configuration File.
Step 1: Create the Code Page Configuration File
The Code Page Configuration File is a YAML file that specifies how Endevor Web Services should handle different character encodings. It helps map elements and metadata to the appropriate code page based on the system, environment, and element characteristics.
Syntax Overview
The YAML file typically defines the following key elements:
- metadata: Specifies the default EBCDIC code page used for metadata like inventory information, CCIDs, and print output.
- default: The default code page for Elements that don’t match any specific rule.
- rules: These are specific code page assignments based on element characteristics, like environment, system, subsystem, and type.
The general structure of the file is:
Example Code Page Configuration File
Here’s an example of a Code Page Configuration File:
In the example above:
- default and metadata are both set to
cp037, an EBCDIC code page.
- The first rule applies to all elements in type
CHDR, specifying that they should use the cp1047 code page.
- The second rule is specific to elements of type
CPGM, which also use cp1047.
- The third rule applies to elements of type
ISPJPN, specifying that they should use the cp290 code page.
Important Notes:
- In YAML file, rules are evaluated in the order they appear. If multiple rules match, the first matching rule takes precedence.
- The wildcards
* and % can be used to apply rules to multiple values. For instance, * matches any value, and % can be used for more specific patterns.
- Quoting all values is recommended, especially if your element names or locations contain special characters.
Step 2: Add the Code Page Configuration File to the Web Services Configuration
Once you’ve created the Code Page Configuration File, the next step is to link it to the Endevor Web Services configuration.
Update the Web Services Configuration File
Update the following XML attribute in the configuration file you configured during the “Create the Web Services Configuration File” phase of the installation.
In the Web Services configuration file, you need to replace the existing CodePage attribute with the CodePageConfig attribute that points to your new YAML file.
For example, if your current configuration file contains the following line:
You will replace it with:
Where codepages.yaml is the name of the YAML file you created. You can either use an absolute path or a relative path from the Web Services configuration directory (usually tomcat/webapps/endevor).
Important Considerations:
- The CodePage and CodePageConfig attributes are mutually exclusive, so make sure not to use both in the same configuration file.
- If you are running multiple Web Services or configurations that access the same Endevor inventory, it’s highly recommended to use a single shared Code Page Configuration File. This helps prevent issues with data corruption, which can occur if a “Retrieve” operation uses one code page, but a subsequent “Update” operation uses another.
Step 3: Test the Configuration
Once the changes are applied, the next step is to test your configuration to ensure that everything is working correctly. Some things to check include:
- Element Encoding: Make sure that the elements are being retrieved, updated, and printed using the correct code page.
- Metadata Handling: Ensure that metadata (like CCIDs and inventory information) is being processed using the
metadata code page (cp037 in our example).
- Rule Matching: Verify that the location-based rules are being applied correctly by checking elements in different environments or with different types.
Prevent Migration Issues
When modifying code page settings for web services, it’s important to be aware of potential element corruption. Elements fetched using an outdated code page setting may become corrupted if re-added to Endevor with a new setting. This is especially critical when working with:
- Git repositories cloned via Endevor Bridge for Git.
- IDE workspaces connected to Endevor through Zowe CLI, Eclipse Plugin, or Explorer for Endevor.
Mitigation Strategies
Here’s how to safeguard your work when updating code page settings:
- Endevor Bridge for Git: For mitigation details, refer to the “Other Considerations” section in the Bridge for Git documentation.
- Explorer for Endevor: Push all changes to Endevor before the update. Afterward, delete and recreate workspaces. For version 1.7.0 or later, workspaces don’t need immediate recreation but will keep the old code page setting until manually refreshed.
- Zowe CLI Endevor Plugin: Same process as Explorer. Push changes first, then delete and recreate workspaces. For version 7.7.2 or later, workspaces will retain the old code page until recreated.
- Endevor Eclipse Plugin: Ensure users push all changes before the update, then delete and recreate their workspaces.
By following these steps, you can prevent element corruption and ensure a smooth transition when changing code page settings.