public class DeployWorkerPackage extends Engine
Modifier and Type | Field and Description |
---|---|
static java.lang.String |
BASE_BUILD_DIRECTORY_URL
The base directory URL for the builds.
|
static java.lang.String |
CLEANUP_PREFIX
Constant suffix for the names of the deployment nodes.
|
static java.lang.String |
COMPLETE_TRANSFORMATION_NAME
The complete TC name for pegasus worker package.
|
static java.lang.String |
COMPLETE_UNTAR_TRANSFORMATION_NAME
The complete TC name for untar.
|
static java.lang.String |
DEFAULT_REFINER
The default transfer refiner name.
|
static java.lang.String |
DEPLOY_WORKER_PREFIX
Constant suffix for the names of the deployment nodes.
|
static java.lang.String |
DERIVATION_NAME
The logical name of the transformation for the worker package
|
static java.lang.String |
DERIVATION_NAMESPACE
The derivation namespace for the worker package.
|
static java.lang.String |
DERIVATION_VERSION
The version number for the derivations for worker package.
|
protected ADag |
mCurrentDag
It is a reference to the Concrete Dag so far.
|
protected java.util.Map<java.lang.String,FileTransfer> |
mFTMap
The FileTransfer map indexed by site id.
|
protected java.lang.String |
mJobPrefix
The job prefix that needs to be applied to the job file basenames.
|
protected java.util.Map<java.lang.String,java.lang.Boolean> |
mLocalTransfers
Map that indicates whether we need local setup transfer jobs for a site or
not.
|
private static java.util.Map<SysInfo.OS,java.lang.String> |
mOSToNMIOSReleaseAndVersion
The map storing OS to corresponding NMI OS platforms.
|
private static java.util.regex.Pattern |
mPattern
Stores compiled patterns at first use, quasi-Singleton.
|
private static java.lang.String |
mRegexExpression
Store the regular expressions necessary to parse the basename from the worker
package url to retrieve the version of pegasus.
|
protected Implementation |
mSetupTransferImplementation
The transfer implementation to be used for staging in the data as part
of setup job.
|
protected java.util.Map<java.lang.String,java.lang.String> |
mSiteToPegasusHomeMap
Maps a site to the the directory where the pegasus worker package has
been untarred during workflow execution.
|
private static java.util.Set<java.lang.String> |
mSupportedOSReleaseVersions
A set of supported OS release and versions that our build process
builds for.
|
protected boolean |
mTransferWorkerPackage
Boolean indicating whether user wants the worker package to be transferred
or not.
|
protected java.lang.String |
mUserSpecifiedSourceLocation
The user specified location from where to stage the worker packages.
|
protected boolean |
mUseUserSpecifiedSourceLocation
Boolean indicating whether to use the user specified location or not
|
protected boolean |
mWorkerNodeExecution
Boolean indicating worker node execution.
|
static java.lang.String |
PACKAGE_NAME
The name of the package in which all the implementing classes are.
|
static java.lang.String |
PATH_VALUE
The path to be set for create dir jobs.
|
static java.lang.String |
PEGASUS_VERSION
The version of pegasus matching the planner.
|
static java.lang.String[][] |
PEGASUS_WORKER_EXECUTABLES
Array storing the names of the executables in the $PEGASUS_HOME/bin directory
Associates the transformation name with the executable basenames
|
static java.lang.String |
POSTSCRIPT_ARGUMENTS_FOR_ONLY_ROTATING_LOG_FILE
The arguments for pegasus-exitcode when you only want the log files to be rotated.
|
static java.lang.String |
REFINER_NAME
The name of the refiner for purposes of error logging
|
static java.lang.String |
TRANSFORMATION_NAME
The logical name of the worker package
|
static java.lang.String |
TRANSFORMATION_NAMESPACE
The transformation namespace for the worker package
|
static java.lang.String |
TRANSFORMATION_VERSION
The version number for the worker package.
|
static java.lang.String |
UNTAR_DERIVATION_NAME
The logical name of the transformation for the untar job.
|
static java.lang.String |
UNTAR_DERIVATION_NAMESPACE
The derivation namespace for the untar job.
|
static java.lang.String |
UNTAR_DERIVATION_VERSION
The version number for the derivations for untar job.
|
static java.lang.String |
UNTAR_PREFIX
Constant suffix for the names of the deployment nodes.
|
static java.lang.String |
UNTAR_TRANSFORMATION_NAME
The logical name of the worker package
|
static java.lang.String |
UNTAR_TRANSFORMATION_NAMESPACE
The transformation namespace for the worker package
|
static java.lang.String |
UNTAR_TRANSFORMATION_VERSION
The version number for the worker package.
|
mBag, mLogger, mLogMsg, mOutputPool, mPoolFile, mPOptions, mProps, mRLIUrl, mSiteStore, mTCFile, mTCHandle, mTCMode, REGISTRATION_UNIVERSE, TRANSFER_UNIVERSE
Constructor and Description |
---|
DeployWorkerPackage(PegasusBag bag)
A pratically nothing constructor !
|
Modifier and Type | Method and Description |
---|---|
ADag |
addCleanupNodesForWorkerPackage(ADag dag)
Adds cleanup nodes in the workflow for sites specified.
|
private TransformationCatalogEntry |
addDefaultTCEntry(java.lang.String site,
java.lang.String pegasusHome,
SysInfo sysinfo,
boolean useFullPath,
java.lang.String name,
java.lang.String executable)
Returns a default TC entry to be used in case entry is not found in the
transformation catalog.
|
protected TransformationCatalogEntry |
addDefaultTCEntryForPegasusWebsite(java.lang.String site,
java.lang.String name)
Returns a default TC entry for the pegasus site.
|
ADag |
addSetupNodes(ADag dag)
Adds a setup node per execution site in the workflow that will stage the
worker node executables to the workdirectory on the sites the workflow
has been scheduled to.
|
private ADag |
addSetupNodesWithoutUntarNodes(ADag dag,
java.util.Set<java.lang.String> deploymentSites)
Adds only the stage worker nodes to the workflow.
|
private ADag |
addSetupNodesWithUntarNodes(ADag dag,
java.util.Set<java.lang.String> deploymentSites)
Adds untar nodes to the workflow, in addition to the stage worker nodes
|
protected java.lang.String |
constructDefaultURLToPegasusWorkerPackage(java.lang.String name,
SysInfo sysinfo)
Constructs the default URL's for the pegasus worker package.
|
private TransformationCatalogEntry |
defaultUntarTCEntry(SiteCatalogEntry site)
Returns a default TC entry to be used in case entry is not found in the
transformation catalog.
|
protected java.lang.String |
getBasename(java.lang.String url)
Returns the basename of the URL using substring.
|
protected java.lang.String |
getCleanupJobname(ADag dag,
java.lang.String site)
It returns the name of the untar job, that is to be assigned.
|
protected java.lang.String |
getDeployJobName(ADag dag,
java.lang.String site,
boolean localTransfer)
It returns the name of the deployment job, that is to be assigned.
|
protected java.util.Set |
getDeploymentSites(ADag dag)
Retrieves the sites for which the deployment jobs need to be created.
|
protected java.lang.String |
getRootDirectoryNameForPegasus(java.lang.String url)
Does regex magic to figure out the version of pegasus from the url, and
use it to construct the name of pegasus directory, when worker package
is untarred.
|
private java.lang.String |
getStagingSite(java.lang.String site)
Returns the staging site for a particular execution site.
|
protected java.lang.String |
getUntarJobName(ADag dag,
java.lang.String site)
It returns the name of the untar job, that is to be assigned.
|
void |
initialize(ADag scheduledDAG)
Initialize with the scheduled graph.
|
static DeployWorkerPackage |
loadDeployWorkerPackage(PegasusBag bag)
Loads the implementing class corresponding to the mode specified by the
user at runtime.
|
protected Job |
makeUntarJob(java.lang.String site,
java.lang.String jobName,
java.lang.String wpBasename)
It creates a untar job , that untars the worker package that is staged
by the setup transfer job.
|
private static java.util.Map<SysInfo.OS,java.lang.String> |
osToOSReleaseAndVersion()
Maps each to OS to a specific OS release for purposes of picking up the
correct worker package for a site.
|
boolean |
runTransferOnLocalSite(Refiner refiner,
java.lang.String site,
java.lang.String destinationURL,
int type)
Returns whether to run a transfer job on local site or not.
|
private static java.util.Set<java.lang.String> |
supportedOSReleaseAndVersions()
A set of OS release and version combinations for which our build
processes build Pegasus binaries.
|
addVector, appendArrayList, complainForHeadNodeURLPrefix, complainForHeadNodeURLPrefix, loadProperties, printVector, stringInList, stringInPegVector, stringInVector, vectorToString
public static final java.lang.String DEPLOY_WORKER_PREFIX
public static final java.lang.String UNTAR_PREFIX
public static final java.lang.String CLEANUP_PREFIX
public static final java.lang.String POSTSCRIPT_ARGUMENTS_FOR_ONLY_ROTATING_LOG_FILE
public static final java.lang.String[][] PEGASUS_WORKER_EXECUTABLES
private static final java.lang.String mRegexExpression
public static final java.lang.String PATH_VALUE
public static final java.lang.String DEFAULT_REFINER
public static final java.lang.String TRANSFORMATION_NAMESPACE
public static final java.lang.String TRANSFORMATION_NAME
public static final java.lang.String TRANSFORMATION_VERSION
public static final java.lang.String UNTAR_TRANSFORMATION_NAMESPACE
public static final java.lang.String UNTAR_TRANSFORMATION_NAME
public static final java.lang.String UNTAR_TRANSFORMATION_VERSION
public static final java.lang.String COMPLETE_UNTAR_TRANSFORMATION_NAME
public static final java.lang.String COMPLETE_TRANSFORMATION_NAME
public static final java.lang.String DERIVATION_NAMESPACE
public static final java.lang.String DERIVATION_NAME
public static final java.lang.String DERIVATION_VERSION
public static final java.lang.String UNTAR_DERIVATION_NAMESPACE
public static final java.lang.String UNTAR_DERIVATION_NAME
public static final java.lang.String UNTAR_DERIVATION_VERSION
public static final java.lang.String PACKAGE_NAME
public static final java.lang.String BASE_BUILD_DIRECTORY_URL
public static final java.lang.String PEGASUS_VERSION
public static final java.lang.String REFINER_NAME
private static java.util.regex.Pattern mPattern
private static java.util.Map<SysInfo.OS,java.lang.String> mOSToNMIOSReleaseAndVersion
private static java.util.Set<java.lang.String> mSupportedOSReleaseVersions
protected ADag mCurrentDag
protected java.lang.String mJobPrefix
protected Implementation mSetupTransferImplementation
protected java.util.Map<java.lang.String,FileTransfer> mFTMap
protected java.util.Map<java.lang.String,java.lang.Boolean> mLocalTransfers
protected java.util.Map<java.lang.String,java.lang.String> mSiteToPegasusHomeMap
protected java.lang.String mUserSpecifiedSourceLocation
protected boolean mUseUserSpecifiedSourceLocation
protected boolean mTransferWorkerPackage
protected boolean mWorkerNodeExecution
public DeployWorkerPackage(PegasusBag bag)
bag
- bag of initialization objectsprivate static java.util.Map<SysInfo.OS,java.lang.String> osToOSReleaseAndVersion()
private static java.util.Set<java.lang.String> supportedOSReleaseAndVersions()
public static DeployWorkerPackage loadDeployWorkerPackage(PegasusBag bag) throws FactoryException
bag
- bag of initialization objectsFactoryException
- that nests any error that
might occur during the instantiation of the implementation.public void initialize(ADag scheduledDAG)
scheduledDAG
- the scheduled workflow.public boolean runTransferOnLocalSite(Refiner refiner, java.lang.String site, java.lang.String destinationURL, int type)
site
- the site handle associated with the destination URL.destURL
- the destination URLtype
- the type of transfer job for which the URL is being constructed.protected java.lang.String getRootDirectoryNameForPegasus(java.lang.String url)
url
- the url.public ADag addSetupNodes(ADag dag)
dag
- the scheduled workflow.private ADag addSetupNodesWithUntarNodes(ADag dag, java.util.Set<java.lang.String> deploymentSites)
dag
- the dagdeploymentSites
- the sites for which the worker package has to be deployedprivate ADag addSetupNodesWithoutUntarNodes(ADag dag, java.util.Set<java.lang.String> deploymentSites)
dag
- the dagdeploymentSites
- the sites for which the worker package has to be deployedpublic ADag addCleanupNodesForWorkerPackage(ADag dag)
dag
- the workflowprotected java.util.Set getDeploymentSites(ADag dag)
dag
- the dag on which the jobs need to execute.protected java.lang.String getDeployJobName(ADag dag, java.lang.String site, boolean localTransfer)
dag
- the workflow so far.site
- the execution pool for which the create directory job
is responsible.localTransfer
- whether the transfer needs to run locally or not.protected java.lang.String getUntarJobName(ADag dag, java.lang.String site)
dag
- the workflow so far.site
- the execution pool for which the create directory job
is responsible.protected java.lang.String getCleanupJobname(ADag dag, java.lang.String site)
dag
- the workflow so far.site
- the execution pool for which the create directory job
is responsible.protected Job makeUntarJob(java.lang.String site, java.lang.String jobName, java.lang.String wpBasename)
site
- the execution pool for which the create dir job is to be
created.jobName
- the name that is to be assigned to the job.wpBasename
- the basename of the worker package that is staged to remote site.private TransformationCatalogEntry addDefaultTCEntry(java.lang.String site, java.lang.String pegasusHome, SysInfo sysinfo, boolean useFullPath, java.lang.String name, java.lang.String executable)
site
- the site for which the default entry is required.pegasusHome
- the path to deployed worker packagesysinfo
- the system information of that site.useFullPath
- boolean indicating whether to use just the basename or
the full pathname
- the logical name of the transformationexecutable
- the basename of the executableprotected TransformationCatalogEntry addDefaultTCEntryForPegasusWebsite(java.lang.String site, java.lang.String name)
site
- the execution site for which we need a matching static binary.name
- logical name of the transformationprotected java.lang.String constructDefaultURLToPegasusWorkerPackage(java.lang.String name, SysInfo sysinfo)
name
- the logical name of the executable, usually worker|binary.sysinfo
- the sysinfo for which the path is required.private TransformationCatalogEntry defaultUntarTCEntry(SiteCatalogEntry site)
site
- the site for which the default entry is required.protected java.lang.String getBasename(java.lang.String url)
url
- private java.lang.String getStagingSite(java.lang.String site)
site
- the execution site.