According to the DevTest 8.0 Scripting Guide ...
"A script in ‘Execute Script (JSR-223)’ test steps does have to supply a specific return value. If no return value is specified the last evaluated expression is taken as the script’s response. This applies to BeanShell, Groovy, and JavaScript."
From the behaviour you're experiencing, it looks like Python doesn't return evaluated expressions. This might include "return", I suppose?
Python scripting isn't supplied with DevTest, so I presume someone at your site found a JSR-223 Python / Jython JAR plug-in and copied it to the DevTest installation.
You could specifically set a property value from within your script:
testExec.setStateValue("mySquareProperty", square(5));
Rick