org.apache.lens.server.query

Class QueryExecutionServiceImpl

java.lang.Object

org.apache.hive.service.AbstractService

org.apache.hive.service.CompositeService

org.apache.lens.server.BaseLensService

org.apache.lens.server.query.QueryExecutionServiceImpl

All Implemented Interfaces:

Externalizable, Serializable, org.apache.hive.service.Service, HealthCheckable, LensService, QueryExecutionService, SessionValidator

public class QueryExecutionServiceImpl
extends BaseLensService
implements QueryExecutionService

The Class QueryExecutionServiceImpl.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	QueryExecutionServiceImpl.FinishedQuery The Class FinishedQuery.
static class	QueryExecutionServiceImpl.QueryCount
static class	QueryExecutionServiceImpl.QueryStatusLogger The Class QueryStatusLogger.
class	QueryExecutionServiceImpl.RewriteEstimateRunnable Chains driver specific rewrite and estimate of the query in a single runnable, which can be processed in a background thread

Nested classes/interfaces inherited from class org.apache.lens.server.BaseLensService

BaseLensService.SessionContext

Nested classes/interfaces inherited from class org.apache.hive.service.CompositeService

org.apache.hive.service.CompositeService.CompositeServiceShutdownHook

Nested classes/interfaces inherited from interface org.apache.hive.service.Service

org.apache.hive.service.Service.STATE

Field Summary

Fields

Modifier and Type	Field and Description
static String	NAME The Constant NAME.
static String	PREPARED_QUERIES_COUNTER The Constant PREPARED_QUERIES_COUNTER.
static String	PREPARED_QUERY_PURGER_COUNTER The Constant PREPARED_QUERY_PURGER_COUNTER.
static String	QUERY_EXPIRY_FAILURE_COUNTER
static String	QUERY_PURGER_COUNTER The Constant QUERY_PURGER_COUNTER.
static String	QUERY_SUBMITTER_COUNTER The Constant QUERY_SUBMITTER_COUNTER.
static String	STATUS_UPDATE_COUNTER The Constant STATUS_UPDATE_COUNTER.
static String	TOTAL_QUERIES_EXPIRED

Constructor Summary

Constructors

Constructor and Description
QueryExecutionServiceImpl(org.apache.hive.service.cli.CLIService cliService) Instantiates a new query execution service impl.

Method Summary

Modifier and Type	Method and Description
boolean	cancelQuery(LensSessionHandle sessionHandle, QueryHandle queryHandle) Cancel the execution of the query, specified by the handle.
void	closeDriverSessions(LensSessionHandle sessionHandle) Allow drivers to release resources acquired for a session if any.
void	closeResultSet(LensSessionHandle sessionHandle, QueryHandle queryHandle) Closes result set by releasing any resources used in serving the resultset.
void	deleteResource(LensSessionHandle sessionHandle, String type, String path)
boolean	destroyPrepared(LensSessionHandle sessionHandle, QueryPrepareHandle prepared) Destroy a prepared query.
QueryCost	estimate(String requestId, LensSessionHandle sessionHandle, String query, LensConf lensConf) Estimate the cost of given query.
QueryHandleWithResultSet	execute(LensSessionHandle sessionHandle, String query, long timeoutMillis, LensConf conf, String queryName) Execute the query with a timeout.
QueryHandle	executeAsync(LensSessionHandle sessionHandle, String query, LensConf conf, String queryName) Asynchronously execute the query.
QueryHandleWithResultSet	executePrepare(LensSessionHandle sessionHandle, QueryPrepareHandle prepareHandle, long timeoutMillis, LensConf conf, String queryName) Execute already prepared query with timeout.
QueryHandle	executePrepareAsync(LensSessionHandle sessionHandle, QueryPrepareHandle prepareHandle, LensConf conf, String queryName) Execute already prepared query asynchronously.
QueryPlan	explain(String requestId, LensSessionHandle sessionHandle, String query, LensConf lensConf) Explain the given query.
QueryPlan	explainAndPrepare(LensSessionHandle sessionHandle, String query, LensConf lensConf, String queryName) Explain the given query and prepare it as well.
QueryResult	fetchResultSet(LensSessionHandle sessionHandle, QueryHandle queryHandle, long startIndex, int fetchSize) Fetch the results of the query, specified by the handle.
List<QueryPrepareHandle>	getAllPreparedQueries(LensSessionHandle sessionHandle, String user, String queryName, String fromDate, String toDate) Returns all the prepared queries for the specified user.
List<QueryHandle>	getAllQueries(LensSessionHandle sessionHandle, String states, String userName, String driver, String queryName, String fromDate, String toDate) Returns all the queries in the specified state, for the given user and matching query name.
List<LensQuery>	getAllQueryDetails(LensSessionHandle sessionHandle, String states, String userName, String driver, String queryName, String fromDate, String toDate) Returns all the queries in the specified state, for the given user and matching query name.
long	getFinishedQueriesCount() Get finished queries count
HealthStatus	getHealthStatus() Returns the health status of the service.
javax.ws.rs.core.Response	getHttpResultSet(LensSessionHandle sessionHandle, QueryHandle queryHandle) Get the http end point for the result set.
long	getInMemoryResultsetTTLMillis() This is the TTL millis for all result sets of type InMemoryResultSet Note : this field is non final and has a Getter and Setter for test cases
EstimatedQueryCollection	getLaunchedQueries() The launched queries.
long	getLaunchingQueriesCount() Get queries being launched count
LensPreparedQuery	getPreparedQuery(LensSessionHandle sessionHandle, QueryPrepareHandle prepareHandle) Get prepared query.
LensQuery	getQuery(LensSessionHandle sessionHandle, QueryHandle queryHandle) Get the query, specified by the handle.
QueryContext	getQueryContext(QueryHandle queryHandle) Gets the query context.
QueryExecutionServiceImpl.QueryCount	getQueryCountSnapshot()
long	getQueuedQueriesCount() Get queued queries count
QueryResultSetMetadata	getResultSetMetadata(LensSessionHandle sessionHandle, QueryHandle queryHandle) Get the result set metadata - list of columns(names and types) and result size.
long	getRunningQueriesCount() Get running queries count
long	getWaitingQueriesCount() Get waiting queries count
void	init(org.apache.hadoop.hive.conf.HiveConf hiveConf)
void	pauseQuerySubmitter(boolean pause) Pause query submitter.
QueryPrepareHandle	prepare(LensSessionHandle sessionHandle, String query, LensConf lensConf, String queryName) Prepare the query.
void	prepareStopping() Prepare stopping.
void	readExternal(ObjectInput in)
void	setInMemoryResultsetTTLMillis(long inMemoryResultsetTTLMillis) This is the TTL millis for all result sets of type InMemoryResultSet Note : this field is non final and has a Getter and Setter for test cases
void	start()
void	stop()
boolean	updateQueryConf(LensSessionHandle sessionHandle, QueryHandle queryHandle, LensConf newconf) Update the query conf.
boolean	updateQueryConf(LensSessionHandle sessionHandle, QueryPrepareHandle prepareHandle, LensConf newconf) Update configuration for prepared query.
void	writeExternal(ObjectOutput out)

Methods inherited from class org.apache.lens.server.BaseLensService

acquire, acquire, closeSession, getCliService, getHiveSessionHandle, getLensConf, getLensConf, getNumberOfSessions, getServerDomain, getSession, getSessionInfo, getSessionManager, getValidPath, openSession, openSession, release, release, removePrefixBeforeURI, restoreSession, validateSession

Methods inherited from class org.apache.hive.service.CompositeService

getServices

Methods inherited from class org.apache.hive.service.AbstractService

getHiveConf, getName, getServiceState, getStartTime, register, unregister

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.lens.server.api.SessionValidator

validateSession

Field Detail

PREPARED_QUERIES_COUNTER

public static final String PREPARED_QUERIES_COUNTER

The Constant PREPARED_QUERIES_COUNTER.

See Also:

Constant Field Values

QUERY_SUBMITTER_COUNTER

public static final String QUERY_SUBMITTER_COUNTER

The Constant QUERY_SUBMITTER_COUNTER.

See Also:

Constant Field Values

STATUS_UPDATE_COUNTER

public static final String STATUS_UPDATE_COUNTER

The Constant STATUS_UPDATE_COUNTER.

See Also:

Constant Field Values

QUERY_PURGER_COUNTER

public static final String QUERY_PURGER_COUNTER

The Constant QUERY_PURGER_COUNTER.

See Also:

Constant Field Values

QUERY_EXPIRY_FAILURE_COUNTER

public static final String QUERY_EXPIRY_FAILURE_COUNTER

See Also:

Constant Field Values

TOTAL_QUERIES_EXPIRED

public static final String TOTAL_QUERIES_EXPIRED

See Also:

Constant Field Values

PREPARED_QUERY_PURGER_COUNTER

public static final String PREPARED_QUERY_PURGER_COUNTER

The Constant PREPARED_QUERY_PURGER_COUNTER.

See Also:

Constant Field Values

NAME

public static final String NAME

The Constant NAME.

See Also:

Constant Field Values

Constructor Detail

QueryExecutionServiceImpl

public QueryExecutionServiceImpl(org.apache.hive.service.cli.CLIService cliService)
                          throws LensException

Instantiates a new query execution service impl.

Parameters:

cliService - the cli service

Throws:

LensException - the lens exception

Method Detail

pauseQuerySubmitter

public void pauseQuerySubmitter(boolean pause)

Pause query submitter. note : used in tests only

init

public void init(org.apache.hadoop.hive.conf.HiveConf hiveConf)

Specified by:

init in interface org.apache.hive.service.Service

Overrides:

init in class org.apache.hive.service.CompositeService

prepareStopping

public void prepareStopping()

Description copied from class: BaseLensService

Prepare stopping.

Overrides:

prepareStopping in class BaseLensService

stop

public void stop()

Specified by:

stop in interface org.apache.hive.service.Service

Overrides:

stop in class org.apache.hive.service.CompositeService

start

public void start()

Specified by:

start in interface org.apache.hive.service.Service

Overrides:

start in class org.apache.hive.service.CompositeService

prepare

public QueryPrepareHandle prepare(LensSessionHandle sessionHandle,
                                  String query,
                                  LensConf lensConf,
                                  String queryName)
                           throws LensException

Description copied from interface: QueryExecutionService

Prepare the query.

Specified by:

prepare in interface QueryExecutionService

Parameters:

sessionHandle - the session handle

query - The query should be in HiveQL(SQL like)

lensConf - The query configuration

queryName - the query name

Returns:

Prepare handle

Throws:

LensException - the lens exception

explainAndPrepare

public QueryPlan explainAndPrepare(LensSessionHandle sessionHandle,
                                   String query,
                                   LensConf lensConf,
                                   String queryName)
                            throws LensException

Description copied from interface: QueryExecutionService

Explain the given query and prepare it as well.

Specified by:

explainAndPrepare in interface QueryExecutionService

Parameters:

sessionHandle - the session handle

query - The query should be in HiveQL(SQL like)

lensConf - The query configuration

queryName - the query name

Returns:

The query plan; Query plan also consists of prepare handle, if it should be used to executePrepare

Throws:

LensException - the lens exception

executePrepareAsync

public QueryHandle executePrepareAsync(LensSessionHandle sessionHandle,
                                       QueryPrepareHandle prepareHandle,
                                       LensConf conf,
                                       String queryName)
                                throws LensException

Description copied from interface: QueryExecutionService

Execute already prepared query asynchronously. Query can be prepared with explain

Specified by:

executePrepareAsync in interface QueryExecutionService

Parameters:

sessionHandle - the session handle

prepareHandle - The QueryPrepareHandle

conf - The configuration for the query to execute

queryName - the query name

Returns:

Returns the query handle

Throws:

LensException - the lens exception

executePrepare

public QueryHandleWithResultSet executePrepare(LensSessionHandle sessionHandle,
                                               QueryPrepareHandle prepareHandle,
                                               long timeoutMillis,
                                               LensConf conf,
                                               String queryName)
                                        throws LensException

Description copied from interface: QueryExecutionService

Execute already prepared query with timeout. Query can be prepared with explain

Specified by:

executePrepare in interface QueryExecutionService

Parameters:

sessionHandle - the session handle

prepareHandle - The QueryPrepareHandle

timeoutMillis - The timeout after which it will return handle, if query did not finish before.

conf - The configuration for the query to execute

queryName - the query name

Returns:

the query handle with result set

Throws:

LensException - the lens exception

executeAsync

public QueryHandle executeAsync(LensSessionHandle sessionHandle,
                                String query,
                                LensConf conf,
                                String queryName)
                         throws LensException

Description copied from interface: QueryExecutionService

Asynchronously execute the query.

Specified by:

executeAsync in interface QueryExecutionService

Parameters:

sessionHandle - the session handle

query - The query should be in HiveQL(SQL like)

conf - The query configuration

queryName - the query name

Returns:

a query handle, which can used to know the status.

Throws:

LensException - the lens exception

updateQueryConf

public boolean updateQueryConf(LensSessionHandle sessionHandle,
                               QueryHandle queryHandle,
                               LensConf newconf)
                        throws LensException

Description copied from interface: QueryExecutionService

Update the query conf.

Specified by:

updateQueryConf in interface QueryExecutionService

Parameters:

sessionHandle - the session handle

queryHandle - the query handle

newconf - the newconf

Returns:

true if update is successful

Throws:

LensException - the lens exception

updateQueryConf

public boolean updateQueryConf(LensSessionHandle sessionHandle,
                               QueryPrepareHandle prepareHandle,
                               LensConf newconf)
                        throws LensException

Description copied from interface: QueryExecutionService

Update configuration for prepared query.

Specified by:

updateQueryConf in interface QueryExecutionService

Parameters:

sessionHandle - the session handle

prepareHandle - the prepare handle

newconf - the newconf

Returns:

true if update is successful, false otherwise

Throws:

LensException - the lens exception

getQueryContext

public QueryContext getQueryContext(QueryHandle queryHandle)

Gets the query context. note: this method is made public to expose it to test cases

Parameters:

queryHandle - the query handle

Returns:

the query context

getQuery

public LensQuery getQuery(LensSessionHandle sessionHandle,
                          QueryHandle queryHandle)
                   throws LensException

Description copied from interface: QueryExecutionService

Get the query, specified by the handle.

Specified by:

getQuery in interface QueryExecutionService

Parameters:

sessionHandle - the session handle

queryHandle - The query handle

Returns:

query status

Throws:

LensException - the lens exception

getPreparedQuery

public LensPreparedQuery getPreparedQuery(LensSessionHandle sessionHandle,
                                          QueryPrepareHandle prepareHandle)
                                   throws LensException

Description copied from interface: QueryExecutionService

Get prepared query.

Specified by:

getPreparedQuery in interface QueryExecutionService

Parameters:

sessionHandle - the session handle

prepareHandle - the prepare handle

Returns:

PreparedQueryContext object

Throws:

LensException - the lens exception

execute

public QueryHandleWithResultSet execute(LensSessionHandle sessionHandle,
                                        String query,
                                        long timeoutMillis,
                                        LensConf conf,
                                        String queryName)
                                 throws LensException

Description copied from interface: QueryExecutionService

Execute the query with a timeout.

Specified by:

execute in interface QueryExecutionService

Parameters:

sessionHandle - the session handle

query - The query should be in HiveQL(SQL like)

timeoutMillis - The timeout after which it will return handle, if query did not finish before.

conf - The query configuration

queryName - the query name

Returns:

a query handle, if query did not finish within the timeout specified else result will also be returned.

Throws:

LensException - the lens exception

getResultSetMetadata

public QueryResultSetMetadata getResultSetMetadata(LensSessionHandle sessionHandle,
                                                   QueryHandle queryHandle)
                                            throws LensException

Description copied from interface: QueryExecutionService

Get the result set metadata - list of columns(names and types) and result size.

Specified by:

getResultSetMetadata in interface QueryExecutionService

Parameters:

sessionHandle - the session handle

queryHandle - the query handle

Returns:

The result set metadata

Throws:

LensException - the lens exception

fetchResultSet

public QueryResult fetchResultSet(LensSessionHandle sessionHandle,
                                  QueryHandle queryHandle,
                                  long startIndex,
                                  int fetchSize)
                           throws LensException

Description copied from interface: QueryExecutionService

Fetch the results of the query, specified by the handle.

Specified by:

fetchResultSet in interface QueryExecutionService

Parameters:

sessionHandle - the session handle

queryHandle - The query handle

startIndex - The start Index from which result rows have to be fetched

fetchSize - Number of rows to be fetched

Returns:

returns the result set

Throws:

LensException - the lens exception

closeResultSet

public void closeResultSet(LensSessionHandle sessionHandle,
                           QueryHandle queryHandle)
                    throws LensException

Description copied from interface: QueryExecutionService

Closes result set by releasing any resources used in serving the resultset.

Specified by:

closeResultSet in interface QueryExecutionService

Parameters:

sessionHandle - the session handle

queryHandle - the query handle

Throws:

LensException - the lens exception

cancelQuery

public boolean cancelQuery(LensSessionHandle sessionHandle,
                           QueryHandle queryHandle)
                    throws LensException

Description copied from interface: QueryExecutionService

Cancel the execution of the query, specified by the handle.

Specified by:

cancelQuery in interface QueryExecutionService

Parameters:

sessionHandle - the session handle

queryHandle - The query handle.

Returns:

true if cancel was successful, false otherwise

Throws:

LensException - the lens exception

getAllQueryDetails

public List<LensQuery> getAllQueryDetails(LensSessionHandle sessionHandle,
                                          String states,
                                          String userName,
                                          String driver,
                                          String queryName,
                                          String fromDate,
                                          String toDate)
                                   throws LensException

Description copied from interface: QueryExecutionService

Returns all the queries in the specified state, for the given user and matching query name.

Specified by:

getAllQueryDetails in interface QueryExecutionService

Parameters:

sessionHandle - the session handle

states - return queries in these state. if null, all queries will be returned. Multiple states can be supplied separated by comma

userName - Get queries submitted by a specific user. If this set to "all", queries of all users are returned

driver - Get queries submitted on a specific driver.

queryName - return queries containing the query name. If null, all queries will be returned

fromDate - start date of time range interval

toDate - end date of the time range interval

Returns:

List of Lens Query object

Throws:

LensException - the lens exception

getAllQueries

public List<QueryHandle> getAllQueries(LensSessionHandle sessionHandle,
                                       String states,
                                       String userName,
                                       String driver,
                                       String queryName,
                                       String fromDate,
                                       String toDate)
                                throws LensException

Description copied from interface: QueryExecutionService

Returns all the queries in the specified state, for the given user and matching query name.

Specified by:

getAllQueries in interface QueryExecutionService

Parameters:

sessionHandle - the session handle

states - return queries in these state. if null, all queries will be returned. Multiple states can be supplied separated by comma

userName - Get queries submitted by a specific user. If this set to "all", queries of all users are returned

driver - Get queries submitted on a specific driver.

queryName - return queries containing the query name. If null, all queries will be returned

fromDate - start date of time range interval

toDate - end date of the time range interval

Returns:

List of query handles

Throws:

LensException - the lens exception

getAllPreparedQueries

public List<QueryPrepareHandle> getAllPreparedQueries(LensSessionHandle sessionHandle,
                                                      String user,
                                                      String queryName,
                                                      String fromDate,
                                                      String toDate)
                                               throws LensException

Description copied from interface: QueryExecutionService

Returns all the prepared queries for the specified user. If no user is passed, queries of all users will be returned.

Specified by:

getAllPreparedQueries in interface QueryExecutionService

Parameters:

sessionHandle - the session handle

user - returns queries of the user. If set to "all", returns queries of all users. By default returns the queries of the current user.

queryName - returns queries matching the query name

fromDate - start time for filtering prepared queries by preparation time

toDate - end time for filtering prepared queries by preparation time

Returns:

List of query prepare handles

Throws:

LensException - the lens exception

destroyPrepared

public boolean destroyPrepared(LensSessionHandle sessionHandle,
                               QueryPrepareHandle prepared)
                        throws LensException

Description copied from interface: QueryExecutionService

Destroy a prepared query.

Specified by:

destroyPrepared in interface QueryExecutionService

Parameters:

sessionHandle - the session handle

prepared - the prepared

Returns:

return true if successful, false otherwise

Throws:

LensException - the lens exception

estimate

public QueryCost estimate(String requestId,
                          LensSessionHandle sessionHandle,
                          String query,
                          LensConf lensConf)
                   throws LensException

Description copied from interface: QueryExecutionService

Estimate the cost of given query.

Specified by:

estimate in interface QueryExecutionService

Parameters:

requestId - the request Id of request used to start estimate operation

sessionHandle - the session handle

query - The query should be in HiveQL(SQL like)

lensConf - The query configuration

Returns:

The query cost

Throws:

LensException - thrown in case of failure

explain

public QueryPlan explain(String requestId,
                         LensSessionHandle sessionHandle,
                         String query,
                         LensConf lensConf)
                  throws LensException

Description copied from interface: QueryExecutionService

Explain the given query.

Specified by:

explain in interface QueryExecutionService

Parameters:

requestId - the request Id of request used to start explain operation

sessionHandle - the session handle

query - The query should be in HiveQL(SQL like)

lensConf - The query configuration

Returns:

The query plan;

Throws:

LensException - the lens exception

deleteResource

public void deleteResource(LensSessionHandle sessionHandle,
                           String type,
                           String path)
                    throws LensException

Throws:

LensException

readExternal

public void readExternal(ObjectInput in)
                  throws IOException,
                         ClassNotFoundException

Specified by:

readExternal in interface Externalizable

Overrides:

readExternal in class BaseLensService

Throws:

IOException

ClassNotFoundException

writeExternal

public void writeExternal(ObjectOutput out)
                   throws IOException

Specified by:

writeExternal in interface Externalizable

Overrides:

writeExternal in class BaseLensService

Throws:

IOException

getHealthStatus

public HealthStatus getHealthStatus()

Returns the health status of the service.

Specified by:

getHealthStatus in interface HealthCheckable

Returns:

getHttpResultSet

public javax.ws.rs.core.Response getHttpResultSet(LensSessionHandle sessionHandle,
                                                  QueryHandle queryHandle)
                                           throws LensException

Description copied from interface: QueryExecutionService

Get the http end point for the result set.

Specified by:

getHttpResultSet in interface QueryExecutionService

Parameters:

sessionHandle - The lens session handle

queryHandle - The query handle

Returns:

returns javax.ws.rs.core.Response object

Throws:

LensException - the lens exception

closeDriverSessions

public void closeDriverSessions(LensSessionHandle sessionHandle)

Allow drivers to release resources acquired for a session if any.

Parameters:

sessionHandle - the session handle

getQueuedQueriesCount

public long getQueuedQueriesCount()

Description copied from interface: QueryExecutionService

Get queued queries count

Specified by:

getQueuedQueriesCount in interface QueryExecutionService

Returns:

queued queries count

getRunningQueriesCount

public long getRunningQueriesCount()

Description copied from interface: QueryExecutionService

Get running queries count

Specified by:

getRunningQueriesCount in interface QueryExecutionService

Returns:

running queries count

getWaitingQueriesCount

public long getWaitingQueriesCount()

Description copied from interface: QueryExecutionService

Get waiting queries count

Specified by:

getWaitingQueriesCount in interface QueryExecutionService

Returns:

waiting queries count

getFinishedQueriesCount

public long getFinishedQueriesCount()

Description copied from interface: QueryExecutionService

Get finished queries count

Specified by:

getFinishedQueriesCount in interface QueryExecutionService

Returns:

finished queries count

getLaunchingQueriesCount

public long getLaunchingQueriesCount()

Description copied from interface: QueryExecutionService

Get queries being launched count

Specified by:

getLaunchingQueriesCount in interface QueryExecutionService

Returns:

Queries being launched count

getQueryCountSnapshot

public QueryExecutionServiceImpl.QueryCount getQueryCountSnapshot()

getLaunchedQueries

@Generated(value="lombok")
public EstimatedQueryCollection getLaunchedQueries()

The launched queries.

getInMemoryResultsetTTLMillis

@Generated(value="lombok")
public long getInMemoryResultsetTTLMillis()

This is the TTL millis for all result sets of type InMemoryResultSet Note : this field is non final and has a Getter and Setter for test cases

setInMemoryResultsetTTLMillis

@Generated(value="lombok")
public void setInMemoryResultsetTTLMillis(long inMemoryResultsetTTLMillis)

This is the TTL millis for all result sets of type InMemoryResultSet Note : this field is non final and has a Getter and Setter for test cases