Source Repository

Lens uses git for its code repository. The repository is available at https://git-wip-us.apache.org/repos/asf/lens.git.

If you are comfortable working in github environment by forking a github repo, sothat you can push changes to your repository before they are accepted in apache, we have a mirror of source at https://github.com/apache/lens. Its better to add the apache repo as remote than github repo, because github repo might be delayed as it is a mirror of apache repo.

Build tools

• A Java Development Kit. You can use java7 and java8.
• Generating site and javadoc with java8 is not supported yet. See [Enunciate issue Enunciate issue and Javadoc issue for more details.
• Apache maven (3.x+)

  Ensure all the tools are installed by executing mvn, git and javac respectively.

  As the Lens builds use the external Maven repository to download artifacts, Maven needs to be set up with the proxy settings needed to make external HTTP requests. The first build of every Lens project needs internet connectivity to download Maven dependencies.

• Be online for that first build, on a good network
• See Maven proxy settings

Integrated Development Environment (IDE)

You are free to use whatever IDE you prefer or your favorite text editor. Note that:

• Building and testing is often done on the command line or at least via the Maven support in the IDEs.
• Set up the IDE to follow the source layout rules of the project.

Building Lens from Source

Download Apache Lens source release from here.

   unzip apache-lens-<version>-source.release.zip
   cd apache-lens-<version>
   mvn clean package -DskipTests

OR Clone Apache Lens source code from https://git-wip-us.apache.org/repos/asf/lens.git

   git clone https://git-wip-us.apache.org/repos/asf/lens.git
   cd lens
   mvn clean package -DskipTests

Once one of the above sets of commands completes successfully, the build will produce lens-dist/target/apache-lens-<version-bin/apache-lens-version-bin/server>, lens-dist/target/apache-lens-<version-bin/apache-lens-version-bin/client>, and lens-dist/target/apache-lens-<version-bin/apache-lens-version-bin/ui>. These can be used as the Lens server installation directory, Lens client installation directory, and Lens UI installation directory respectively. run lens server, lens client and UI.

Lens UI is written in nodejs, and can be run directly from the module itself. In fact, it's recommended to run from there since the npm dependencies are already fetched in lens-ui module by mvn clean package step above.

   cd lens-ui
   npm start

For making changes in lens-ui code, it can be started in dev mode where it'll watch for your changes and re-bundle whenever changes are made. To run ui server in dev mode, use the following command:

   npm run dev

The build will also produce debians for server, client and ui in lens-dist/target. Client debian uses /usr/local/lens/client as the Lens client installation directory and Server debian uses /usr/local/lens/server as the Lens server installation directory. UI debian uses /usr/local/lens/ui as the installation directory.

Apache Lens depends on Hive. Please build Hive from Source or install it using the documentation here. After installing Lens and Hive, refer here for running lens client and lens server from installation directories.

Building Hive from Source

   git clone https://github.com/apache/hive.git

   cd hive

   git checkout <release-tag>

   mvn clean package -DskipTests -Pdist,deb

Once above package command completes successfully, packaging/target will have apache-hive-$project.version-bin. This build also produces source, binary tar.gz files and deb package for hive.

Set the environment variable HIVE_HOME to point to the Hive installation directory built from source:

  cd packaging/target/apache-hive-$project.version-bin/apache-hive-$project.version-bin
  export HIVE_HOME=`pwd`

Code compliance

All contributions should satisfy the following requirements

• All public classes and methods should have informative javadoc comments.
  • Do not use @author tags.
• All existing unit tests and integration tests should pass.
• New unit tests should be provided to demonstrate bugs and fixes. Lens uses TestNG test framework. If any code change does not include unit test, the contributor should give the reason why it is not possible to include a unit test.
• Project documentation corresponding to the change should be updated along with the code change. See Documentation section to know how Lens documentation is organized and how to update
• Code must be formatted according to java standards, with the following changes:
  • Trailing White spaces: Remove all trailing white spaces.
  • Indentation: Never use tabs! Always use 2 space indents.
  • Line wrapping: Always use a 120-column line width.
• Use slf4j framewrok for logging and use parameterized logging. Avoid commons logging and log4j fully, and those should be removed from transitive dependencies of newer dependencies added.
• All working files (java, xml, others) should have the ASF license header in all versioned files.
• If new features requires illustrative examples, they should be added in lens-examples.

Naming convention for configuration properties

Developers should follow the following naming convention for configuration properties

• All server configuration names start with lens.server.
• All configuration overridable for each query start with lens.query.
• All configuration of drivers start with lens.driver. For HiveDriver the names start with lens.driver.hive. and for JDBCDriver it is lens.driver.jdbc
• All configuration overridable for each session start with lens.session.

Generate a patch

   git status

  git add src/.../MyNewClass.java
  git add src/.../TestMyNewClass.java

  git diff > LENS-1234.patch

  git diff master..HEAD > LENS-1234.patch

  git apply --check lens_patch.patch
  git apply lens_patch.patch

Review request

Install reviewboard command line tool:

• Make sure you have python installed and easy_install is available.
• $ easy_install -U setuptools
• More help available at Reviewboard Documentation.

  rbt post --parent=master

  git diff master..HEAD > ~/LENS-##.patch

  rbt post -u --parent=master

  rbt close --close-type={submitted/discarded} #####(request id)

Submit patch

After the patch is reviewed on review board and validated by committers, they will post a "Ship It!" review. After that, you're expected to take the patch from reviewboard, attach it on the jira and make the jira "Patch Available". Only after that's done, will it be committed.

Unifying the above

So The above steps can get time consuming after few times. We recommend you use the above steps at least a few times to understand and internalize the process. There is also an automated way of doing the above things. One of the lens developers has developed a command line tool for it. It's available at https://github.com/prongs/rbt-jira. The typical contributor workflow will be like following:

• Make Changes
• Generate patch using git diff
• Post patch to reviewboard.
• Review Happens
• Go back to step 1 to make changes according to the review. Or go to next step if Ship It! is provided
• Download patch from review board
• Attach patch in jira
• Make jira patch available
• Still a review can cause the patch to be cancelled. Go back to step 1. Or next step if it's committed.
• Done.

So there's multiple steps involved in loops. The rbt-jira tool aims to automate precisely them. The documentation of the tool you can check on it's own github, as it's still in development and is subject to change.

Project documentation

The design, architecture and feature documentation needs to be updated in src/site of parent module. The documentation is organized into four menus under the site.

Configuration documentation

Configuration pages should be linked from user guide and admin guide with respect to the configuration exposed to them. See Developer FAQ on how to update config docs.

REST api documentation

REST api documentation is auto generated through Enunciate. Once the javadoc for the resource api is updated correctly, the REST api doc should get updated.

If a new resource is added in lens-server module, it should be updated in lens-server/enunciate.xml sothat the REST api doc gets generated. If a new module is added with resources and the module pom entry need to updated with enunciate plugin usage and tools/scripts/generate-site-public.sh needs to be updated with site generation and publishing the docs.

Feature documentation

End users of Lens include :

• Querying users : mostly un-aware of system details, and un-aware of underlying data layout.
• Schema designer : Schema designer would understand the data model and come up with schema for their data.
• Server administrator : Lens server administrator understands the server setup, how multiple execution engines can be added/removed and how to support multiple storage.
• Developer : Developers can develop new drivers, new features on server or client or anything to do with code.

  Whenever a new feature is added to Lens, the developer should understand to whom the feature is applicable and put it in proper menu - "User Menu" for Querying users and schema designers; "Admin Menu" for server administrators and "Developer Menu" for developer related features.

  The following details about the feature should be documented :

• The use case : Explaining Why the feature is required.
• The feature itself : What the feature is.
• How to enable the feature, if any.
• Who are the users of feature and who are not, if any.
• Illustration with example would be very welcome

  The design documentation related to the feature can go in developer documentation or Design docs. It would be necessary to add in which version the feature is available if the documentation is in confluence. Also when behavior is modified or improved on existing features, version tagging is quite useful. Any defaults (in terms of config or behavior) assumed with the feature should also be highlighted. Configuration descriptions should be linked to config apt files, so that they are always in sync with code.

Confluence usage

Cwiki should be used for documentation that cannot go into code, which is some adhoc documentation. This falls into following main categories

• Design doc for a feature, which is just proposed or it is under implementation. Once the feature is implemented doc should be updated in project documentation
• Project roadmap
• Discussions : Placeholder for discussions that cannot be done in jira
• Presentations : Links to slideshare or google docs
• Events : Developer/User Meetup minutes

  Cwiki should not be used for documentation that is already present in code, which is the following

• Architecture documentation
• Feature documentation
• How to contribute/commit/release pages
• Generated doc for REST API/Javadoc
• Getting started pages

How to update documentation?

The new doc files can added in src/site of parent module. Or doc change can be done in existing files under src/site. Update the config files and run the TestGenerateConfigDoc test calss for updating the config docs. Once the changes are done, contributor can run the below command.

  mvn site:run

This will start localhost doc server on http://localhost:8080, which can opened through browser and doc can be validated.

How to update the config docs?

Add/Delete/Modify the config to the resource config files(Ex: lens-client-default.xml file for Client resource). Run the TestGenerateConfigDoc, which automatically updates the config apt files under src/site.

How to update CLI doc?

So cli doc is auto generated by reading all the annotations in cli java files. So if you've added a new cli command or modified an existing command, please make sure to have enough documentation in the help section of the command's annotations. Look at other commands to get an idea. After any modifications, run TestGenerateCLIUserDoc to automatically re-generate cli doc. Once you do that, start the site locally and verify the change is reflected at http://localhost:8080/user/cli.html. Also verify it doesn't look out of place by looking at neighbouring command documentations.

How to add license headers for newly added files?

Run the command mvn license:format. This is add license headers for all the files automatically. If some files need to excluded they should be put in excludes section in parent project pom's license-maven-plugin.

How to check all licenses are fine?

Run the command mvn apache-rat:check. If check needs to be excluded for any file, it should be put in excludes section in parent project pom's apache-rat-plugin.

What is versioning strategy in Lens?

Lens follows three number versioning which is major.minor.revision. If the current release is 2.0.0, the next usual development release will be 2.1.0. If there needs to be separate release on released version and not from development branch (usually critical patch release), it will be 2.0.1. If the next release is not compatible with previous release, then the major version needs to be incremented, then it would become 3.0.0. This way all 2.x.x releases will be compatible with one another. And incompatibility is clearly communicated to users through major version number change.

The jira fix version for all the issues in 2.0.x is called 2.0, and 2.1.x is called 2.1. For patch releases from release branch, the jira fix version can be exact patch release version number.

What is the branching strategy in Lens?

Lens has two main branches - master and current-release-line. All the day to day development happens on master branch. current-release-line branch is used to make releases. When master branch is ready for release (all improvements/features/bug fixes marked for release are fixed and all tests passing), master will be merged to current-release-line. The version number on master will be incremented to next development version. The only issues that can be cherry-picked into current-release-line till release is rolled are critical/blocker bugs, documentation changes and test case changes. Once all the issues are verified for the release, and release will be triggered from current-release-line.

If a critical release (not pulling code from master) needs to be made, a new branch will be created with release number, by checking out current-release-line branch. And changes will be put on the branch. Once the branch is ready they will merged to current-release-line and released. The changes should be cherry-picked back into master from current-release-line once the release is made and resolving conflicts in master if any. Having two main branches makes all release tags to be created on current-release-line branch and removes the pile up of old and stale branches, which are created by one for each release.

For major version increments, current-release-line will be branched to a a $major.x-line and current-release-line and master will be moved to next major version.

There can be feature branches created from master, if feature is not actively developed in master branch directly. For a feature branch to be created a contirbutor can start discuss thread on dev list for consensus on whether it is required.

How to add a new Driver in Lens? Please refer to this wiki article. For examples you can refer to the following review board requests 1. Druid driver in Lens 2. Elastic Search Driver

How to add a new Error Code in Lens? Please refer to this wiki article