Measure

A measure is a quantity that you are interested in measuring. Measure is a field having a default aggregator, a format string, unit, start time and end time. Can also have min and max value.

Dim Attribute

Dim Attributes are not measured, they are more like properties of your data. e.g. Location, user name etc.

• A dim attribute can be as simple as having name and type. // Base Dim Attribute
• A dim attribute can also refer to a dimension // Referenced Dim Atrribute
• A dim attribute can be associated with an expression // Expression Column
• A dim attribute can be a hierarchy of dim attributes. // Hierarchical Dim Attribute
• A dim attribute can define all the possible values it can take. // Inline Dim Attribute

Expression Column

Expression column has one or many expression specs. So you can declare one expression field is specified by one expression for some time, and another for other times.

Join Chain

A join chain is a directional path between two conceptual tables. So if a conceptual table t1 has a chain jc to conceptual table t2, t1 can access t2's fields by saying jc.<t2_field_name>.. A join chain consists of one or more Join paths. A path is defined by sequence of edges where an edge is defined like table1.some_field=table2.some_field. In a path, end table of one edge should be same as start table of next edge.

Dimension

A Dimension is a a conceptual table which only contains dim attributes, expressions and join chains.

Cube

A cube is a conceptual table which contains dim attributes, measures, expressions and join chains.

Cubes are of two types:

Dimension tables

Dimension Tables are associated with Dimensions. They can be available in multiple storages.

Fact Tables

The fact table is associated with cube, specified by name. Fact can also be available in multiple storages. The fact will be used to answer the queries on derived cubes as well. Typically facts will belong to only base cubes and derived cubes will inherit all the facts of the base cube.

Dim storage tables

Dimension storage table is the physical dimension table for the associated storage. Dimension storage table can have snapshot dumps at specified regular intervals or a table with no dumps, so the storage table can have zero or one update period.

If the dimension storage table is being updated regularly, older partitions are expected to have lesser data than latest partitions. Examples could be, country id to country name mappings. Newer partitions are supposed to contain at least equal --- or, possibly more --- number of mappings than older partitions. Once a partition is registered, all the older partitions become obsolete.

So in accordance with this, while registering partition, lens registers an additional partition with value latest which has path same as the actual latest partition. So promoting that dim storage tables are always supposed to be queried with latest partition. This is reflected in lens's query translation logic where only latest partition is queried.

Since only one partition is relevant for dim storage tables, lens maintains a hash map for quicker lookup of latest partition.

Fact storage tables

Unlike dim storage tables, all partitions in fact storage tables are relevant and queryable. So there is no latest partition. Instead, lens maintains something called Partition Timeline. They are better explained in this wiki page

Here we'll explore some of the things that you need to be aware of to interact with timelines as a lens user.

Timelines are stored in storage table's properties, which is again cached in memory. Since one fact storage table can have multiple update periods and partitions registered for them can be different, there is need to have timelines for all update periods. Also one storage table can have multiple partition columns. So timelines need to be present for all partition columns too. So for one fact storage table, if x is number of update periods and y is number of partition columns, there will be x*y timelines for it.

You can see the current timeline of the fact by this rest api

Alternatively, on cli you can view like this:

lens-shell>fact timelines --fact_name sales_aggr_fact2
EndsAndHolesPartitionTimeline(super=PartitionTimeline(storageTableName=mydb_sales_aggr_fact2, updatePeriod=DAILY, partCol=dt, all=null), first=2015-04-12, holes=[], latest=2015-04-12)
EndsAndHolesPartitionTimeline(super=PartitionTimeline(storageTableName=mydb_sales_aggr_fact2, updatePeriod=DAILY, partCol=ot, all=null), first=2015-04-12, holes=[], latest=2015-04-12)
EndsAndHolesPartitionTimeline(super=PartitionTimeline(storageTableName=mydb_sales_aggr_fact2, updatePeriod=DAILY, partCol=pt, all=null), first=2015-04-13, holes=[], latest=2015-04-13)
EndsAndHolesPartitionTimeline(super=PartitionTimeline(storageTableName=local_sales_aggr_fact2, updatePeriod=HOURLY, partCol=dt, all=null), first=2015-04-13-04, holes=[], latest=2015-04-13-05)
EndsAndHolesPartitionTimeline(super=PartitionTimeline(storageTableName=local_sales_aggr_fact2, updatePeriod=DAILY, partCol=dt, all=null), first=2015-04-11, holes=[], latest=2015-04-12)

lens-shell>fact timelines --fact_name sales_aggr_fact2 --storage_name local
EndsAndHolesPartitionTimeline(super=PartitionTimeline(storageTableName=local_sales_aggr_fact2, updatePeriod=HOURLY, partCol=dt, all=null), first=2015-04-13-04, holes=[], latest=2015-04-13-05)
EndsAndHolesPartitionTimeline(super=PartitionTimeline(storageTableName=local_sales_aggr_fact2, updatePeriod=DAILY, partCol=dt, all=null), first=2015-04-11, holes=[], latest=2015-04-12)

lens-shell>fact timelines --fact_name sales_aggr_fact2 --storage_name local --update_period HOURLY
EndsAndHolesPartitionTimeline(super=PartitionTimeline(storageTableName=local_sales_aggr_fact2, updatePeriod=HOURLY, partCol=dt, all=null), first=2015-04-13-04, holes=[], latest=2015-04-13-05)

lens-shell>fact timelines --fact_name sales_aggr_fact2 --storage_name local --update_period HOURLY --time_dimension delivery_time
EndsAndHolesPartitionTimeline(super=PartitionTimeline(storageTableName=local_sales_aggr_fact2, updatePeriod=HOURLY, partCol=dt, all=null), first=2015-04-13-04, holes=[], latest=2015-04-13-05)
lens-shell>

Any time you feel that the timeline is out of sync with the actual partitions registered, just set cube.storagetable.partition.timeline.cache.present = false in the storage table's properties and restart lens server. Now this will read all partitions registered for the storage table and re-create the timeline. After creation, it'll update table properties to reflect the correct value.