I don't know if there's any specific upgrade path for that and would have to defer on that to our SAP/Syclo experts active on this forum.
However, one thing is certain - you cannot run SAP WM 5.3 on SMP 3.0. You have to upgrade the app to SAP Work Manager 6.1. This involves an upgrade of SMERP and SMFND components in SAP ECC, new deployment files for SMP 3.0 and new Agentry clients. As for any enhancements you might have done in release 5.3, you'll have to port them to the new version. The document here: https://websmp106.sap-ag.de/~sapdownload/012002523100010362502014E/SAPWM60_EnhancementsGuide.pdf
describes how to move the enhancements from the older versions to version 6.0. While this document pertains to SMP 2.3, most of the concepts described there are still applicable in 6.1.