JSAGA

Hi Tünde,

Thank you for this detailled description, it contains all the information I needed to reproduce and understand your problems with FileTransfer.

Answer to your questions about CREAM and LCG-CE:
When the FileTransfer attribute is not supported by the middleware (LCG-CE) or not yet implemented by the adaptor (CREAM), JSAGA tries to transfer files through the job input/output streams, which are available only in interactive mode...
I think that this is better than just throwing a NotImplemented exception as it should, but this mechanism still needs to be improved. I will try to think about a better solution and let you know on this thread if I find one...

Answer to your questions about WMS:
Output is obtained only with jsaga-job-get-output_WMS.sh => that's the expected behavior for batch mode.
File is transferred to /tmp/output.txt => I have opened a bug for this:
https://forge.in2p3.fr/issues/show/454

Regards,
Sylvain

Hi Sylvain,
Thank you for your reply.
Now I have another question about submitting to WMS.
If I submit a batch job, but don't specify the Output argument, then after the job is done, I get a message that the output is retrieved, but there is actually nothing to retrieve. Shouldn't there be a default 'stdout' file to retrieve or state that there is nothing to retrieve?

$ jsaga-job-run.sh  -b -Executable /bin/hostname -Queue trekker.nikhef.nl:2119/jobmanager-pbs-short 
-r wms://graszode.nikhef.nl:7443/glite_wms_wmproxy_server

$jsaga-job-run.sh -d -b -Executable /bin/hostname -Queue trekker.nikhef.nl:2119/jobmanager-pbs-short \
-r wms://graszode.nikhef.nl:7443/glite_wms_wmproxy_server 
Type = "Job";
Executable = "/bin/hostname"; 
Requirements = true
&& (other.GlueCEStateStatus=="Production")
&& other.GlueCEUniqueID == "trekker.nikhef.nl:2119/jobmanager-pbs-short" ;
Rank = (-other.GlueCEStateEstimatedResponseTime);
OutputStorage = "/tmp";

$ jsaga-job-get-output_WMS.sh [wms://graszode.nikhef.nl:7443/glite_wms_wmproxy_server]
    -[https://grasveld.nikhef.nl:9000/q48jIWM9q53cKsuS5ZiElA]  
Job output have been retrieved successfully 

Hi Tünde,

We can not add a default 'stdout' file to retrieve because it would not be conform to the SAGA specification.

We could add a state to specify that there is nothing to retrieve since our cleanup method is not standard anyway, but when such a method will be added to the SAGA specification it will probably not have this state.

The proper way to do this would be to query the job description, but retrieving the job description of a job from a JVM different than the JVM that submitted the job is not yet supported by JSAGA. Until this will be supported, we can replace the message with "Job output have been retrieved successfully (if it exists)" and let the user remember what he asked for.

Hi Tünde,

Management of attributes FileTransfer, Input, Output and Error has been improved in JSAGA core engine, in order to be homogeneous for all the plug-ins and more conform to the SAGA specification.
Support for FileTransfer has also been implemented in CREAM plug-in, and rewritten in WMS plug-in (I have closed bug https://forge.in2p3.fr/issues/show/454).
Finally, the behavior of jsaga-job-run command has been modified in order to enable to submit jobs that are neither batch jobs (i.e. no --batch option), nor interactive jobs (i.e. add -Input, -Output or -Error option).

These modifications should solve all the problems you mentioned in this thread, except the last one (jsaga-job-output command still returns message "Job output have been retrieved successfully (if it exists)").

Please let me know if you still have problems with this new version.

Hi,
I know that the following command shouldn't work, but I accidentally forgot to redirect the standard output and I still tried to get the output then I got something like this:

$ jsaga-job-run.sh -d -b -Output output.txt -Executable /bin/date \
-r wms://graszode.nikhef.nl:7443/glite_wms_wmproxy_server
Type = "Job";
Executable = "/bin/date";
Requirements = true
&& (other.GlueCEStateStatus=="Production");
Rank = (-other.GlueCEStateEstimatedResponseTime);
  StdOutput = "output.txt";
OutputStorage = "/tmp";

$ jsaga-job-run.sh -b -Executable /bin/date -Output output.txt \
-r wms://graszode.nikhef.nl:7443/glite_wms_wmproxy_server
[wms://graszode.nikhef.nl:7443/glite_wms_wmproxy_server]-[https://grasveld.nikhef.nl:9000/qhATthRoLX0kMAyTgIZ29Q]

$ jsaga-job-get-output.sh [wms://graszode.nikhef.nl:7443/glite_wms_wmproxy_server]-[https://grasveld.nikhef.nl:9000/qhATthRoLX0kMAyTgIZ29Q]
Exception in thread "main" java.lang.NullPointerException
        at fr.in2p3.jsaga.impl.job.instance.AbstractSyncJobImpl.doPostStagingAndCleanup(AbstractSyncJobImpl.java:305)
        at fr.in2p3.jsaga.impl.job.instance.AbstractSyncJobImpl.postStagingAndCleanup(AbstractSyncJobImpl.java:294)
        at fr.in2p3.jsaga.command.JobGetOutput.main(JobGetOutput.java:62)

Best,
Tünde

Hi Tünde,

Please note that according to the SAGA specification, jsaga-job-get-output should (and will) retrieve nothing.

However, the bug you found also occurred when using the FileTransfer attribute.

It is now fixed in the new snapshot.

Best regards,
Sylvain