The support for version 7 of Metastorm BPM is nearly at an end. Many customers have version 7 systems. Many even have version 6 systems still running. These will probably need to be migrated to Metastorm BPM 9. As well as allowing continued support, this allows supporting platform software that also has run out of support life to be discarded in favour of newer supported software.
The real problem many are facing is working out how to manage this. As a provider of Metastorm BPM services and solutions for 12 years, we have a performed large number of migrations between Metastorm BPM versions. In recent years, these have all been conversions from Metastorm BPM 6 and 7 to Metastorm BPM 9.
Migration to Metastorm BPM 9 is not as simple as previous versions. Effectively, the platform itself has been completely altered. Although Metastorm did a remarkable job of making the Designer and Client look and work at least similarly to previous versions, under the hood it is a very different beast. Because of this, and to allow a staged migration, the system cannot be updated 'in situ' as before, but must be migrated to a new system.
This migration is a two-part migration. Firstly the code is migrated, and then validated and fixed to work in version 9. Then the data itself can be migrated. When complete, this can then all be tested and fixed as needed. This is not a simple process. If tackled correctly, it should be successful. If it is not, you can waste a lot of time, effort and money fixing mistakes that have been made.
In this article we outline some of the concepts that you need to consider, and how you can plan for your successful migration. In our next article in this section, we will discuss the available alternatives.
Things to consider
Before developing plans, it is best to consider several aspects of the migration and new system.
Consider if the processes will all be migrated at one time, or separately. If separately, consider whether they will run 'side by side' or in separate environments.
You will need to make sure that all dependant Process are migrated at the same time if you are performing a staged migration.
Make sure that all stakeholders understand that there must be a code-freeze in the current system until migration is complete. As this can take several months, it is important that it is understood right from the start. If there are any changes required for any reason, make sure that it is understood that all changes to the existing system will need to be applied and tested in the new system as well.
Consider if you will be making use of new functionality. If you are interested in the multi-language capability, for example, you should set this up when installing, as it is much simpler at that point.
Other considerations may be to simplify or redevelop processes or forms. Of course, this will involve analysis and design tasks, but these may be quicker and more effective than testing and fixing existing functionality that has become over-complicated.
This is the expected structure of any of our migration plans. This has also been laid out in a detailed Process Design using the Business Optix Author:
This gives you a better overall view of the process itself, and the various inputs and deliverables from each activity. It also adds the roles and responsibilities, not discussed here.
The Business Optix Author allows you to create detailed Process maps, and a variety of other models. It also includes a fully-featured collaboration library that allows users to view, comment on, rate and approve models. Please note that much of the functionality is not available in this model as it is not within our library. If you are interested in trying it out for free, please go here:
We shall be publishing a review and 'Quick Start' guide shortly.
Develop Test Cases & Test Plans
It is essential to develop a proper Test Plan for your migration. If there is no proper plan, then the migration will likely fail, even if in small and non-critical ways. The overall effect will probably cause a reduction in confidence of the system, and even the platform itself (Metastorm BPM).
A well-designed and well-executed Test Plan will very likely prevent this and promote confidence.
Test Cases should be well thought out. The most valuable resources when building these are probably expert users of the system. Find out what problems have occurred in the past, and what scenarios are likely to generate problems. From these discussions, test cases could be generated.
Another important consideration that is often overlooked is test cases for data. The existence and accuracy of at least sample sets of data should be checked. Reports compared between systems can be very effective to manage this.
It will be essential to get training in the use of the Metastorm BPM version 9 Development environment. We can deliver a highly-regarded Transition Couse. Feel free to contact us here:
Pre-Migration Code Review
Before the file migration, the existing version 7 code should be examined for potential changes and improvements that are not as straightforward after the file migration.
It may seem obvious, but business users need to be aware that the code will have to freeze during the code migration phase. This means no new features. If anything needs to be updated to fix faults or meet legislative changes, everybody needs to be aware that the changes will need to be applied and tested in the version 7 code, and again in the version 9 code.
One thing we have noticed is that the version 9 Designer can struggle when there are a lot of components in a Project (previously called a Procedure). It is definitely wise to consider breaking up large Procedure files before migration as well, as Projects will be migrated as a whole. Having smaller Projects will also give you more chances of finding and fixing any faults during both file and data migration.
If any components are shared, it is well worth putting them in a library accessible to all necessary Projects. Components such as Flags and Roles are probably the most likely candidates.
This should be done before the files are migrated. After migration, there can be significant issues managing the renaming of Projects.
The most important thing to do is to make sure that all grids are named appropriately. If, like us, you were a bit slack at naming grids, you will end with a whole load of meaningless names for Business Objects. Naming the grids will ensure that Business Objects are also sensibly named after migration, making debugging and maintenance a great deal easier.
Make sure your grid columns are using the correct case. In version 7 and before, the code managing this was case-insensitive. In version 9, just about everything is case-sensitive. This is possibly only important when using Oracle as a database, as it always returns uppercase names by default regardless of the columns in the select statement used.
Note that Flag parameters are now strongly typed in Version 9. In previous versions you could pass in different parameters and types for each flag call. You can no longer do this in version 9, so you need to make sure Flag parameters are consistent across the system for each Flag.
Now the code gets migrated. This involves use of the File Migration tool.
Save all the libraries and procedure files.
For all file migrations we recommend not to create large solutions with many Libraries and/or Projects. Ideally, in our experience, you should have one Project or Library per Solution. In some rare instances, it may be appropriate to have a Library and a Project. There is no benefit to grouping Projects in a Solution.
Migrate all the libraries first. These will need to be validated and deployed. If they cannot be deployed, the migration of any Procedure files relying on these libraries will not work.
Now migrate the Procedures. Remember to keep the 'Mappings' file. There is also an HTML file produced explaining any renaming or other changes performed. This could be essential, although it is typically rather lengthy.
In each File Migration, you will need to be aware of several faults in the generated systems. We will not be publishing this here, but will be willing to provide consulting on the issues you will need to be aware of.
Post-Migration Code Review
After the file migration, the migrated code should be examined for potential changes and improvements.
Processes are generally the same in version 9 as previous versions. The one difference is the use of Sub-Processes (previously Map Segments). The data for these is stored in a separate table. You may need to consider how that will affect any data lookups or reports you may have.
You may be able to make use of the new 'only show action if' property of Actions. This can reduce the need for complex dynamic roles that we had to employ to hide or show Actions under certain conditions.
Make use of Local variables where necessary. These variables exist for the life of the form, and will not be saved to the database. Using these can allow you to remove any number of redundant 'temporary' variables you may have been forced to create in previous versions.
You need to be aware that some Forms will need to be reformatted. There are two main changes to fields that can force this. The first is the space required for an asterisk before required fields. This can move a caption further to the left, potentially overlapping other fields. The second is that grid column titles and rows are taller. This can make grids difficult to navigate until they are expanded.
You may be able to make use of the new 'only show form if' property of Forms. This can reduce the need for complex dynamic roles that we had to employ to hide or show Forms under certain conditions.
Note that all code that relies on string matching will now be case-sensitive. In previous versions it was all case-insensitive. In version 7 it is also possible to use the 'greater than' and 'less than' comparisons for strings. This is no longer possible in as it is not possible in C#. You would have to add your own function or find a different way to achieve your goal.
Any code that references Metastorm internal data may need updating. The most common of these are likely to be Process (Map), Stage, Action and Role names. All of these may need to be updated as spaces are replaced with underscores. Note that there may be other renaming due to naming conflicts (eg all actions have to have a unique name) and reserved words.
Note also that some replaced features will not work as they did in version 7 and earlier. We do have a few functions in our library that replicates these built-in functions while fixing them.
Most server scripts we have created are in Jscript. This works well, as the transition from Metastorm Jscript to C# is not complex. The code structure is similar, although it is obviously a 'real' language with more stringent rules, and is also very strongly typed.
VBScript is not so easy to convert to C#, and you have no alternative to doing this manually. Developers familiar with both languages are useful if this is required.
Jscript.Net is probably the easiest to convert to C#. Although the parameter passing construct was exceedingly awkward in JScript.net server scripts in Metastorm BPM 7, the language itself is very similar to C#.
From our experience, many client scripts will need to be, or would certainly benefit from being, rewritten. If you have many of these, and they are in any way complex, you should consider this carefully.
If you access any part of the Metastorm BPM form using any method that is not part of the standard supplied and supported functions, you will definitely have to rewrite these.
Given the much more extensive functionality in Metastorm BPM 9, it is often worthwhile abandoning Client Scripts in favour of real code. There are still a few instances where these are still required, such as setting cookies and copying text to the clipboard.
If you have any 'dynamic SQL' grids, these need special consideration. These are typically used for query forms, and involve building the SQL in code to get the required results. Most of these will require Scripted Business Objects to be built. These are not intuitive, and are code-intensive. They are very powerful, however.
Across all the code, you should see how much database access and updating is performed in your system. If there is a lot, for example to update read-only grids or even for dropdown option lists, you may need to edit quite a lot of the migrated code.
To make best use of the Metastorm BPM 9 database access functionality, you would also need to parameterise these where possible.
You need to be aware that any SQL that references Metastorm internal data may need updating. The most common of these are likely to be Process (Map), Stage, Action and Role names. All of these may need to be updated as spaces are replaced with underscores. Note that there may be other renaming due to naming conflicts (eg all actions have to have a unique name) and reserved words.
As noted in the Forms and Processes sections, some Roles may be simplified or even removed by using the new 'only show if' functionality for both Forms and Actions.
This point is a very good time to consider alternative designs and additional functionality. Metastorm BPM 9 has a very different architecture to Metastorm BPM 6 or 7. In this section, we list the various main areas of interest.
You may be able to take advantage of the use of Captions to replace Stage and Action names. These can contain characters that were not allowed in version 7, for example. You can also give two Stages or Actions the same caption, although this can potentially become confusing, so must be used carefully.
Forms are very similar in version 9, but they do have some very important new features. These mainly involve panels, which can be docked, and the fact that all fields can be resizable as the Form size itself is changed. You may want to discuss how they can be taken advantage of once the migration is complete.
You may be able to take advantage of the use of Form Captions to replace Form names. These can contain characters that were not allowed in version 7, for example. You can also give two Forms the same Caption.
The proprietary code format of Metastorm BPM 4, 5, 6 and 7 has been dubbed 'Meta-script' by many. Most of us know that this can become unwieldy at times, and may well look forward to a neater and more functional code structure that a language such as C# can offer. As a C++ and Java developer myself, I have found the transition welcome and easy. Many may find that getting to grips with a real programming language a bit daunting. Concepts such as classes, casting, loops and use of arrays are not simple concepts. Some of the errors generated by incorrect syntax can be highly confusing, for example.
Since we had no ability to create reusable functions (except using server scripts) in Metastorm BPM 7 and before, we tended to duplicate a lot of code. We have also seen this many times in systems written by others. It is very much worth seeing how much can be made reusable if you have any duplication.
Most of the changes allow a far greater use of reusable code and shared components. To make best use of these, you may want to consider redesigning the way code is managed.
As an example, we now recommend having all code for a Project in Server Scripts. This prevents the need for hunting through myriads of different components to find references. Since all functions can easily be promoted to The Visual Script Toolbox, you can simply add these to each event to replace code. You could then never have to open the event for any component again. This also gives you much better scope for collaborative development between Process Designers and Developers in the future. See details in our Training Courses.
If you make use of Web Services, you can make a lot of use of the new Connections in version 9. These allow you to use Web Services much more effectively. Most, if not all, of the existing code to manage these may be redundant now.
Common Business Objects
Consider the number of grids you have, and how complex they are. If you have a great many, you will end up with a large number of Business Objects. It is worth considering how much of this SQL it will be possible to reuse in common Business Objects.
Database access in version 9 can be very much simplified in a number of ways. The most prominent of these is the use of editable Business Objects. We often read from and update tables other than the main process table during key processes. In order to do this in version 7, we used to have to select multiple fields, assign values, and then write values back to the database.
In version 9, it is quite possible to use any Business Object as the main source of data in a form, and even in a process. This can be as simple as setting the default parameter, and then using the fields from the Business Object in a form directly, or even in process events. Values assigned to these Business Object variables will be saved when the activity completes. You would not need to do anything at all in terms of SQL to maintain this data, as this is all handled at the Business Object level.
The first test is to validate the code. This can throw up some confusing errors, so you will need time and patience. The error reporting in version 9 has come under a lot of fire for being unhelpful and difficult to understand. Often every single reference throws an error, so you have hundreds of them, which is almost useless. We agree that this is probably the weakest feature of Metastorm BPM 9, and one that needs to be fixed as soon as possible.
When you have eliminated all the errors generated by the validation, you can attempt to deploy. This may throw up quite different errors. It appears that the validation in the Designer is different to the validation in the Deployment service. In some cases this is understandable, but in others it is not.
Once the Projects are deployed, you can start the data migration. This allows the data migration to be tested on the current Production data, and gives you an opportunity to test the functionality with existing data.
This will typically be managed using the Data Migration tool. This can be done one Process at a time, but you will need to make sure that all dependant Process are migrated at the same time if you are performing a staged migration.
Manual Data Migration
In some processes, it is not possible to migrate the data using the Data Migration tool. This may be because of errors thrown by the tool, or perhaps because of the size of the data and attachments for a process. In some cases, it may be possible, but will just take too long. We had one system that was still attempting to migrate after 48 hours. As we needed the full migration from version 6 to version 9 to occur over a weekend, we were forced to migrate the data manually.
It is important that this is tested as soon as possible so that support can be sought to resolve errors, and any such plans can be made as early as possible. A manual data migration is exceedingly complex, and should be avoided at all costs. Identifying issues early on can help to ensure this is not required. If it is required, we have performed several manual data migrations, and could provide advice or assistance.
There may be issues with the data after migration. It is not possible to specify all possible issues, but we have not yet come across any system where nothing needed to be updated.
It is important to keep all of the SQL used to update data, and keep these as scripts to execute after the full data migration.
Test Data Migration
Now that we have both the code and the data migrated, we can start testing the functionality.
This seems as though it may be overkill, but we have found that the best way to test functionality of processes is to perform a set of actions in version 7, then to try the same activity in version 9 with the same data.
In one system we migrated, the business users actually set up the data migration test to occur just before their regular monthly cycle of activity. They then ran the exact same actions in the version 9 system just as they were being executed in the version 7 system, against the same data but migrated. The result was a very thorough set of tests without the need for extensive Test Cases. Bear in mind that Test Cases will still be required, this won't catch everything!
Finally, once the test data is all migrated, you can start on the test cases you previously generated. Note that quite different errors may be generated. It is worth noting that the errors reported in the Designer log may be difficult to fathom too. Along with validation, these really need to be made at least as good as they were in version 7 to be properly useful.
Once the Code and Data are both validated and tested and are deemed accurate, you can then plan the final phase where the data is migrated from the existing system to the new system. Obviously the code should not need any more work, but the data will have been updated since you started the code migration and data migration testing.
Note that we recommend this entire process is performed as a 'dry-run' to test that it works as planned. There may be many involved parties, and many steps and resources involved. It is a good idea to act this out, taking timings all the way. This then ensures the process will work, and that you can plan the times well.
Note that as some tasks may take many hours, these should be set to be started overnight if time is constrained. This can ensure a minimal down-time of the system.
Create new system & database
Consider whether you will want multi-language capabilities now or in the future. If so, it is much better to set this up right from the start rather than add it later.
Deploy all the version 9 libraries and projects to the new system. Note that the Engine must be running for this.
Shut down the version 9 Engine and Deployment services. Make sure these are set to 'Manual' start, just in case the server is rebooted, even accidentally.
On the Day
Shut down the version 7 Engine service. Make sure this are set to 'Manual' start, just in case the server is rebooted, even accidentally.
Migrate data as planned
Migrate the data as done previously.
Execute SQL scripts
Execute al the SQL scripts generated in the data migration and subsequent tests.
Perform any tests on the data that you have generated to ensure that the data has been migrated successfully.
Start Version 9
Start up the version 9 engine. Check that folders open, and blank forms and actions can be started. There should be no need to submit any actions to perform this test.
Examine Designer Log & Event log
Examine the Designer Log and Event Log for any errors. There may be issues such as system actions that have failed that are not otherwise evident.