Introduction

Typical data layout in a data ware house is to have fact data rolled up with time and reduce dimensions at each level. Fact data will have dimension keys sothat it can be joined with actual dimension tables to get more information on dimension attributes.

The typical data layout is depicted in the following diagram.

Lens provides an abstraction to represent above layout, and allows user to define schema of the data at conceptual level and also query the same, without knowing the physical storages and rollups, which is described in below sections.

Metastore model

Metastore model introduces constructs Storage, Cube, Fact table and Dimension table, Partition.

Metastore API

LENS provides REST api, Java client api and CLI for doing CRUD on metastore.

Query Language

User can query the lens through OLAP Cube QL, which is subset of Hive QL.

Here is the grammar:

  CUBE SELECT [DISTINCT] select_expr, select_expr, ...
  FROM cube_table_reference
  [WHERE [where_condition AND] [TIME_RANGE_IN(colName, from, to)]]
  [GROUP BY col_list]
  [HAVING having_expr]
  [ORDER BY colList]
  [LIMIT number]
  
  cube_table_reference:
  cube_table_factor
  | join_table
  join_table:
  cube_table_reference JOIN cube_table_factor [join_condition]
  | cube_table_reference {LEFT|RIGHT|FULL} [OUTER] JOIN cube_table_reference [join_condition]
  cube_table_factor:
  cube_name or dimension_name [alias]
  | ( cube_table_reference )
  join_condition:
  ON equality_expression ( AND equality_expression )*
  equality_expression:
  expression = expression
  colOrder: ( ASC | DESC )
  colList : colName colOrder? (',' colName colOrder?)*
  
  TIME_RANGE_IN(colName, from, to) : The time range inclusive of ‘from’ and exclusive of ‘to’.
  time range clause is applicable only if cube_table_reference has cube_name.
  Format of the time range is <yyyy-MM-dd-HH:mm:ss,SSS>

OLAP Cube QL supports all the functions that hive supports as documented in Hive Functions

Query engine provides following features :

• Figure out which dimension tables contain data needed to satisfy the query.
• Figure out the exact fact tables and their partitions based on the time range in the query. While doing this, it will try to minimize computation needed to complete the queries.
• If No Join condition is passed for Joins,join condition will be inferred from the relationship definition.
• Projected fields can be added to group by col list automatically, if not present and if the group by keys specified are not projected, they can be projected
• Automatically add aggregate functions to measures specified as measure's default aggregate.

  Various configurations available for running an OLAP query are documented at OLAP query configurations

Query API

LENS provides REST api, Java client api, JDBC client and CLI for doing submitting queries, checking status and fetching results.