Globus environment

Step. 0 source $GLOBUS_LOCATION/etc/globus-user-env.sh

Step. 00 export X509_CERT_DIR=/home/leesangm/.globus/certificates

Step. 000 Check if this works,
globusrun -o -r grid-co.ncsa.teragrid.org/jobmanager '&(executable=/bin/ls)'

condor_rm -forcex -all : force the immediate local removal of jobs in the X state

Condor/Birdbath documentations

Birdbath section in the condor manual:
http://www.cs.wisc.edu/condor/manual/v6.8/4_4Application_Program.html
Marlon's Blog
Condor Installation: http://communitygrids.blogspot.com/2007/07/beating-condor-like-rented-buzzard.html
Submitting Globus job through condor: http://communitygrids.blogspot.com/2006/09/submitting-globus-job-through-condor.html

Condor: Birdbath APIs javadoc

After few days of suffering, I generated this javadoc from condor birdbath related source codes. I got the source code from,
(1) birdbath jar file: download from http://www.cs.wisc.edu/condor/birdbath/birdbath.jar
(2) type classes generated from WSDL (birdbath1.jar): included in the condor6.8.6 release (/lib/webservice)
http://www.cs.indiana.edu/~leesangm/ogce/condor/docs/

VLAB

mvn clean install cargo:deploy -Dmaven.test.skip=true -f with_condor/task-executor/pom.xml

FAS Script file for the SC07 Demo

Script file generateDB.sh is designed especially for the demo at SC07. This script manages three accounts, leesangm, mpierce, and yuma. First, all of the information about these users will be removed.Then new information will be registered. If you want to modify information, open generateDB.sh modify it, and run it.

To run this script,
Step 1. You should download this .
All of the library jar files, and script files are included.
Step 2. Run the command.
./generateDB.sh

That's it.

Command Line Access to the FAS0.7

Command line tool provides immediate access to the FAS server. This is very useful for the test runs.
To user this command line tool, you don't need to run the service on your site. You will need only client of the service.

Step 1. You should download this .
All of the example, script files are included.

Step 2. Run the command.
(1)To create new user space
./run.sh AddNewUserSpace -f [FAS service location] -l [username] -h [host name] -p [port number] -dn [DN of the community cert for this host] -pr [protocol] -us [logical space for this user]
e.g.) ./run.sh AddNewUserSpace -l leesangm -h "gridfarm001.ucs.indiana.edu" -p "2811" -pr "gridftp" -us "/home/quakesim/"

(2)To query user space
./run.sh QueryUserSpace -f [FAS service location] -l [username] -h [host name] -p [port number] -dn [DN of the community cert for this host] -pr [protocol] -us [logical space for this user]
e.g.) ./run.sh QueryUserSpace -l leesangm

(3)To remove user space
./run.sh RemoveUserSpace -f [FAS service location] -l [username] -h [host name] -p [port number] -dn [DN of the community cert for this host] -pr [protocol] -us [logical space for this user]
e.g.) ./run.sh RemoveUserSpace -l leesangm

[NOTE]
Default value of FAS service location is

http://gridfarm002.ucs.indiana.edu:8080/axis2/services/FileAgentService

Default value of DN of the Community Certificate is Teragrid Quakesim cert.
/C=US/O=National Center for Supercomputing Applications/CN=QuakeSim Community User

File Agent Service V0.7

1. Introduction
File Agent Serivce is a web service providing access to the information about community user space. This user space is accessed with Teragrid community certificate. This version provides 7 operations including 3 operations from the previous version.

• queryDataSpaceInfo
• registerDataSpaceInfo
• removeDataSpaceInfo
• createDataSpace
• queryUserSpaceConvention
• registerUserSpaceConvention
• removeUserSpaceConvention

Section 2 will discribe how to access the service, and each of the operations are described in the section 3.

2. Access to the service
Required jar file:
-FileAgentService.aar and basic jar file for axis2.
Download tarball [48MB contains all the lib and source code].
Service is now up and running on,
http://gridfarm002.ucs.indiana.edu:8080/axis2/services/FileAgentService
The WSDL file is available at,
http://gridfarm002.ucs.indiana.edu:8080/axis2/services/FileAgentService?wsdl
For detail code example, please refer the sample code.

3. Operations
3.1 queryDataSpaceInfo
This operation provides query interface to the data space information. Input parameter is query statement described in WSDL and output parameter is the query result.
Here is how to build an input query for this operation. First, dataspace information includes following items,

• username : user's portal id
• host: remote location of the data resource
• port: port number of the data resource
• commCertDN: DN of the community certificate that used for this resource
• protocol: protocol to access this resource
• LogicalUserSpace: logical user space assigned for the user

Above items are accessible through getter/setter methods of the FASBean object that is included in the FileAgentService.aar file.

For example,
FASBean fb = new FASBean();
fb.setUsername("fiona");
fb.setHost("gridfarm002");
fb.setPort("123");
fb.setCommCertDN("nydb");
fb.setProtocol("gridftp");
fb.setLogicalUserSpace("/home/communityspace/fiona");

For the queryDataSpaceInfo operation, you can build very simple query without knowing any of complicated query language. Specify items that you know exactly. Leave items as null if you are not sure. This operation will process the null item as a wildcard in the query.

For example, if you want to get all the dataspace information about user A in the host GGG, then set the username and host and leave everything else as null. (Just don't set the value).

3.2 registerDataSpaceInfo
This operation registers new data space information. Define FASBean and use it as parameter of this operation. Please refer the sample code.

3.3 removeDataSpaceInfo
This operation removes data space inforamtion. This works very similar to the queryDataSpaceInfo operation. Therefore, if you want to remove all of the information about user A, you should specify username only, and leave all of other itesm null. Please note if you set complete null input parameter to thie operation, it will completely reset your database.

3.4 createDataSpace
[TODO] This operation is almost same with thte registerDataSpaceInfo operation. On top of creating data space information, this operation will access the remote resource and create your's logical space as well.

3.5 queryUserSpaceConvention
This is a utility operation for the advanced users. If the administor wants to organize ther user's logical space with more standard convention, 3.5 ~ 3.7 operations should be very useful.
This operation provides following informations,

• host: host name of the remote resource
• port: port number of the remote resource
• attr: attribution of this convention
• defaultLoc : convention of the user space

For example, let's assume that an administrator should organize users' data under the community space which is /g/d/commUser. This space is hosted by host A and port 1234. Also this host has multiple mounted resources such as /tg/h/j/hpss for the HPSS server, and /sc/datacatacitor for the High performance data capacitor. The user's space can be organized following way.

host: A
port: 1234
attr: High performance data capacitor
defaultLoc: /sc/datacapacitor/userspace/username

host: A
port: 1234
attr: Network File System
defaultLoc: /g/d/commUser/userspace/username

host: A
port: 1234
attr: HPSS
defaultLoc: /tg/h/j/hpss/userspace/username

Based on these informations, when a new user tries to create a data space, the system can query the convention of the defaultLoc and create the user space accordingly. Also the application can show the attribute of the location so that it can provide more precise information instead encrypted directory path.

You can use this operation in very similar way to the queryDataSpaceInfo. Specify all of the items you know exactly and send the request. The operation will process your query as predicates combined as AND predicates and process non-specified items as wildcard items.

3.6 registerUserSpaceConvention
This operation registers new User space convention. Section 3.5 explains how to organize the community user space using this operation. For more information, please refer the sample code.

3.7 removeUserSpaceConvention
This operation removes user's space convention. For more information, please refer the sample code.

Appendix. Sample Code
Download tarball [48MB contains all the lib and source code]
Sample Code Directory: src/org/ogce/fileagentservice/client
AddNewUserSpace.java
QueryUserSpace.java
RemoveUserSpace.java
ConventionExample.java

Community Proxy Single sign-on

This is about the sigle sign-on process with the community proxy certificate for the OGCE porttal (Gridsphere version). As soon as the user's login name and password are verified the portal service accesses teragrid proxy server, gets the community proxy certificate and stores it with user's login name in the proxy manager.

Step 0. prepare the account
-- get community account
-- http://grid.ncsa.uiuc.edu/myproxy/tgsso.html

Step 1. change the code
org.gridlab.gridsphere.impl.service.core.user.LoginServiceImpl
-- access the proxy server (teragrid proxy server)
-- get a proxy cert
-- register to the proxy manager with the currrent login name

Step 2. compile and generate jar
On gridsphere directory,

ant gridsphere-jar

*make sure that the detendent jar files are located in the lib directory.
proxymanager-api-4.2.jar
cog-jglobus-1.2-050621.jar
jgss-1.0.jar

*make sure that the classpath is pointing the lib directory in the compile task.

Step 3. copy jar to current OGCE deployment under,
/shared/lib/

File Agent service access from portlet: compile

STEP 0: Shut down tomcat

STEP 1: add 3rd party jar for axis2,
axis2-adb-1.3.jar
axis2-kernel-1.3.jar
axiom-api-1.2.5.jar
annogen-0.1.0.jar
axiom-api-1.2.5.jar
axiom-dom-1.2.5.jar
axiom-impl-1.2.5.jar
commons-codec-1.3.jar
commons-httpclient-3.0.1.jar
stax-api-1.0.1.jar
wsdl4j-1.6.2.jar
wstx-asl-3.2.1.jar
XmlSchema-1.3.2.jar
FileAgentService-0.5.aar
command:
ogce-portal-only/maven-2.0.7/bin/mvn install:install-file -DgroupId=axis2 -DartifactId=axis2-adb -Dversion=1.3 -Dfile="axis2-adb-1.3.jar" -Dpackaging=jar
ogce-portal-only/maven-2.0.7/bin/mvn install:install-file -DgroupId=axis2 -DartifactId=axis2-kernel -Dversion=1.3 -Dfile="axis2-kernel-1.3.jar" -Dpackaging=jar
ogce-portal-only/maven-2.0.7/bin/mvn install:install-file -DgroupId=FileAgentService -DartifactId=FileAgentService -Dversion=0.5 -Dfile="FileAgentService-0.5.aar" -Dpackaging=jar
ogce-portal-only/maven-2.0.7/bin/mvn install:install-file -DgroupId=axis2 -DartifactId=axiom-api -Dversion=1.2.5 -Dfile="axiom-api-1.2.5.jar" -Dpackaging=jar

Step 2. Add s to the project.pom file


axis2
axiom-api
1.2.5
compile


axis2
axiom-dom
1.2.5
compile


axis2
axiom-impl
1.2.5
compile



axis2
axis2-adb
1.3
compile



axis2
annogen
0.1.0
compile



axis2
axis2-kernel
1.3
compile



axis2
commons-codec
1.3
compile



axis2
commons-httpclient
3.0.1
compile


axis2
stax-api
1.0.1
compile


axis2
wsdl4j
1.6.2
compile


axis2
wstx-asl
3.2.1
compile


axis2
XmlSchema
1.3.2
compile



FileAgentService
FileAgentService
0.5
compile



STEP 3: compile and install
On the directory of ogce-portal-only,
maven-2.0.7/bin/mvn clean install -f portlets/comp-file-management/pom.xml

STEP 4: try it.
./tomcat-startup.sh

Organizing information about Community proxy

Well..if a person wants to transfer a file from her community data repository to her personal/remote data file system through the file-comp-management portlet, what do we need to provide? That person will need to access the community proxy to access her community data repository. At a same time, she should access her personal proxy which is relevant to the remote resource. Do we have to ask her every time which proxy she is gonna use? What if it's non-interactive task such as job processing with mix-and-match proxies?

Since remote access to the resource requires the location of the resource, I think about simple but nice organizing of the information about community proxies and personal proxies based on the location. For each of the users, if {Username, Host, Port} is provided after the user issues proxies, ProxyManager will let me find the proxy for the host with the username.

Also if the virtulization of the user space for the community repository is considered, I may have to add the pointer to the user space specified by the system. If it is the community repository based on the Unix file system and the system relocates each of the user's space as /CommunityRepository/[username], than userspace will set as /CommunityRepository/myusername. If it is not a community repository then "~" can simply lead the user to the user's home directory that is specified by the Unix file system. Therefore, now I have, {Username, Host, Port, Userspace}.

Hang on... on top of this, I have to consider about the users who have both of the community acess and personal access to the same resource. For the huge data repository, this can happen quite often. I might have to add some identifier of the proxy, so that Userspace is not confusing. This doesn't have to be universally unique but it should be unique enough within the set of proxies for the same host and user. User's DN which is available from the GSSCredential object can be a good candidate. Now triplet is grown as {Username, Host, Port, Userspace, ProxyIdentifier}.

What else can be added....?

Test note for the ProxyManager Portlet

1. Test Items

• Create a personal proxy
• Create a community proxy
• Set default proxy
• Get a new personal proxy but not create a copy of existing community proxy
• Romove proxy
• Test with 3rd party user

2. FixMe list
3. Possible addition
display whether it's a community proxy.

Test note for the Comp-file-management portlet.

1. Test list

• File list
• GoTo Directory
• Go UpperDirectory
• Upload
• Transfer
• Home
• Rename

2. FixMe List
(1) After transfer or upload file, file list displays home directory. - Fixed.
(2) After the transfer the file, the size of the file is listed as 0.
(3) Last Modified of the destination File Listing is not human-friendly.

Commands for using maven2

Installing 3rd party jar
maven-2.0.7/bin/mvn install:install-file -DgroupId=proxymanager-api-4.2.1 -DartifactId=proxymanager-api-4.2.1 -Dversion=4.2.1 -Dfile="/home/leesangm/ogce-portal-only/proxymanager-api-4.2.1.jar" -Dpackaging=jar

Modify comp-file-management portlet to use Cog Abstraction APIs

Comp-file-management portlet is implemented using the JSR0168 specification, and allows a user to list/browse files on a source and/or destination resource. Also it provides features to upload, and download user's files. User can transfer files between remote resources. Users upload local file to the remote resource as well.
Currently, major gridFTP functionality is implemented by access to the gridFTP APIs. To modify the classes to use COG Abstraction APIs, some of the classes from the grid port package and comp-file-management portlet are re-written.
*All of the added classes are packaged under .utils because of the confliction between the package names.

Added/Modified Classes

1. edu.tacc.gridport.portlets.interactive.utils.FileListingManager

Origin: GridPort

New Package: edu.tacc.gridport.portlets.interactive.utils
Modified Methods:

FileListingBean executeFileListing(ActionRequest req, ActionResponse res, FileListingBean currentListing)
ArrayList constructFileBeans(Collection files, String currentDir)
ArrayList constructFileBeans(FileInfo[] files, String currentDir)

Added Methods:

FileResource getConnect(GSSCredential proxy, String host, String port,String purpose, boolean remotePassive,String provider)

FileResource getConnect(PortalRequest request, String host, String port,String purpose, boolean remotePassive, String provider)
String getUpperDirectory(String currentDirectory)
String getPortalUsername(PortletRequest request)
void putIntoSession(PortletRequest request, String name, Object obj)
String constructPermissionString(Permissions user, Permissions group, Permissions all, boolean dir)

2. edu.tacc.gridport.portlets.interactive.utils.FileUploadManager
Origin: Grid Port
New Package: edu.tacc.gridport.portlets.interactive.utils
Modified Methods:
void uploadFileToRemote(ActionRequest req)

3.edu.tacc.gridport.portlets.interactive.helpers.FileTransfer
Origin: edu.tacc.gridport.portlets.interactive.helpers (comp-file-management portlet)
New Package: no change
Added Methods:
void fileUploadFromUserMachine(String toHost, String localFileFullName, String remoteFileFullName, GSSCredential credential)

Modified Methods:

void thirdPartyTransfer(String fromHost, String fromFileFullName,String toHost, String toFileFullName, GSSCredential credential)

Compile and deploy single portlet

New version (Sep. 07, 2007)
mvn clean install -f portlets/filetransfer/pom.xml

Old version

1. Goto /ogce/ogce-portal/ogce-portal
2. ../third_party_tools/maven-1.0.2/bin/maven gp:deploy-portlet -Dportlet.name=filetransfer -Denv.CATALINA_HOME=../portal_deploy/apache-tomcat-5.5.12
   

Note: use maven V.1.xx
source: http://collab-ogce.blogspot.com/2007/07/re-compiling-individual-portlets.html

setup the globus credential

setup the globus credential info:(http://manacar.blogspot.com/2006/12/myproxy-guidelines-for-community-grids.html)