@@ -1,2 +1,4 @@

+^Meta$

+^doc$

 ^.*\.Rproj$

 ^\.Rproj\.user$

@@ -1,3 +1,5 @@

+Meta

+doc

 .DS_Store

 **/.DS_Store

 .Rhistory

@@ -6,4 +8,4 @@

 .Rapp.history

 .RData

 .*.zip

-*.Rproj

\ No newline at end of file

+*.Rproj

@@ -35,7 +35,8 @@

 #'

 #' @details

 #' This function works only with datatset or GRangesList all whose samples or 

-#' Granges have the same region coordinates (chr, ranges, strand)

+#' Granges have the same region coordinates (chr, ranges, strand) ordered in 

+#' the same way for each sample 

 #' 

 #' In case of GRangesList data input, the function searches for metadata

 #' into metadata() function associated to GRangesList.

@@ -52,10 +53,14 @@

 #' filter_and_extract(test_path, region_attributes = c("pvalue", "peak"))

 #' 

 #' ## This statement imports a GMQL dataset as GRangesList and filters it 

-#' ## including at output only "pvalue" and "peak" region attributes

+#' ## including at output only "pvalue" and "peak" region attributes, the sort

+#' ## function makes sure that the region coordinates (chr, ranges, strand) 

+#' ## of all samples are ordered correctly

+#' 

 #' 

 #' grl = import_gmql(test_path, TRUE)

-#' filter_and_extract(grl, region_attributes = c("pvalue", "peak"))

+#' sorted_grl = sort(grl)

+#' filter_and_extract(sorted_grl, region_attributes = c("pvalue", "peak"))

 #'

 #'

 #' @export

deleted file mode 100644

@@ -1,303 +0,0 @@

-title: "RGMQL: GenoMetric Query Language for R/Bioconductor"

-author: 

-- "Simone Pallotta" 

-- "Marco Masseroli"

-date: "2017-11-14"

-bibliography: bibliography.bib

-output: BiocStyle::pdf_document

-vignette: >

-    %\VignetteIndexEntry{Vignette Title}

-    %\VignetteEngine{knitr::rmarkdown}

-    %\VignetteEncoding{UTF-8}

-link-citations: true

-

-# Introduction

-

-Recent years have seen a tremendous increase in the volume of data generated 

-in the life sciences, especially propelled by the rapid progress of 

-Next Generation Sequencing (NGS) technologies. 

-This high-throughput technologies can produce billions of short DNA or RNA 

-fragments in excess of a few terabytes of data in a single run.

-Next-generation sequencing refers to the deep, in-parallel DNA sequencing 

-technologies providing massively parallel analysis and extremely 

-high-throughput from multiple samples at much reduced cost. 

-Improvement of sequencing technologies and data processing pipelines 

-is rapidly providing sequencing data, with associated high-level features, 

-of many individual genomes in multiple biological and clinical conditions. 

-To make effective use of the produced data, the design of big data algorithms 

-and their efficient implementation on modern high performance 

-computing infrastructures, such as clouds, CPU clusters 

-and network infrastructures, is required in order to achieve scalability 

-and performance. 

-For this purpose the GenoMetric Query Language (GMQL) has been proposed 

-as high-level, declarative language to process, query, 

-and compare multiple and heterogeneous genomic datasets for biomedical 

-knowledge discovery [@Bioinformatics2015]

-

-## Purpose

-

-A very important emerging problem is to make sense of the enormous amount and 

-variety of NGS data becoming available, i.e. to discover how different genomic 

-regions and their products interact and cooperate with each other. 

-To this aim, the integration of several heterogeneous DNA feature data 

-is required.

-Such big genomic feature data are collected within numerous and 

-heterogeneous files, usually distributed within different repositories, 

-lacking an attribute-based organization and a systematic description 

-of their metadata. 

-These heterogeneous data can contain the hidden answer to very important 

-biomedical questions.

-To inveil them, standard tools already available for knowledge extraction 

-are too specialized or present powerful features, but have a rough interface 

-not well-suited for scientists/biologists.

-GMQL addresses these aspects using cloud-based technologies 

-(including Apache Hadoop, mapReduce, and Spark), and focusing on genomic data 

-operations written as simple queries with implicit iterations over thousands 

-of heterogeneous samples, computed efficiently [@IEEE7484654].

-This RGMQL package makes easy to take advantage of GMQL functionalities also 

-to scientists and biologists with limited knowledge of query and 

-programming languages, but used to the R/Bioconductor environment. 

-This package is built over a GMQL scalable data management engine 

-written in Scala programming language, released as Scala API [@githubrepo] 

-providing a set of functions to combine, manipulate, compare, and extract 

-genomic data from different datasources both from local and remote datasets.

-These functions allow performing complex GMQL processing and queries without 

-knowledge of GMQL syntax, but leveraging on R idiomatic paradigm and logic.

-

-

-# Genomic Data Model

-

-The Genomic Data Model (GDM) is based on the notions of datasets 

-and samples[@modeling2016] 

-Datasets are collections of samples, and each sample consists of two parts, 

-the region data, which describe portions of the genome, and the metadata, 

-which describe sample general properties and how observations are collected.

-In contrast to other data models, it clearly divides, and comprehensively 

-manages, observations about genomic regions and metadata.

-GDM provides a flat attribute based organization, just requiring that 

-each dataset is associated with a given data schema, which specifies 

-the attributes and their type of region data.

-The first attributes of such schema are fixed (chr, start, end, strand); 

-they represent the genomic region identifying coordinates.

-In addition, metadata have free attribute-value pair format.

-

-## Genomic Region 

-

-Genomic region data describe a broad variety of biomolecular aspects and are 

-very valuable for biomolecular investigation.

-A genomic region is a portion of a genome, qualified by a quadruple of values 

-called region coordinates:

-$$< chr, left, right, strand >$$

-Regions can have an arbitrary number of associated values, according to 

-the processing of DNA, RNA or epigenomic sequencing reads that determined 

-the region.

-

-## Metadata

-

-Metadata describe the biological and clinical properties associated with 

-each sample.

-They are usually collected in a broad variety of data structures and formats 

-that constitute barriers to their use and comparison GDM models metadata 

-simply as arbitrary semi-structured attribute-value pairs, 

-where attributes may have multiple values.

-

-## Genomic Sample

-

-Formally, a sample s is a collection of genomic regions modeled as 

-the following triple: $$< id, {< r_i,v_i >}, {m_j} >$$ where:

-

-* id is the sample identifier

-* Each region is a pair of coordinates $r_i$ and values $v_i$

-* Metadata $m_j$ are attribute-value pairs $< a_j,v_j >$

-

-Note that the sample id attribute provides a many-to-many connection between 

-regions and metadata of a sample.

-Through the use of a data type system to express region data, and of arbitrary 

-attribute-value pairs for metadata, GDM provides interoperability across 

-datasets in multiple formats produced by different experimental techniques.

-

-## Dataset

-

-A dataset is a collection of samples uniquely identified, with the same region 

-schema and with each sample consisting of two parts:

-

-* region data: describing characteristics and location of genomic portions

-* metadata: expressing general properties of the sample

-

-Each dataset is typically produced within the same project by using the same 

-or equivalent technology and tools, but with different experimental 

-conditions, described by metadata.

-

-Datasets contain large number of information describing regions of a genome, 

-with data encoded in human readable format using plain text files.

-

-GMQL datasets are materialized in a standard layout composed of three 

-types of files:

-

-1. genomic region tab-delimited text files with extension .gdm, or .gtf 

-if in standard GTF format

-2. metadata attribute-value tab-delimited text files with the same fullname 

-(name and extension) of the correspondent region file and extension .meta

-3. schema XML file containing region attribute names and types

-

-All these files reside in unique folder called files.

-

-<!-- ![GMQL dataset folder](dataset_gmql.png) -->

-

-In RGMQL package dataset files are considered read-only.

-Once read, genomic information is represented in abstract structure inside 

-the package, mapped to a R GRanges data structure at occurency.

-

-

-# GenoMetric Query Language

-

-The GenoMetric Query Language name stems from the language ability to deal 

-with genomic distances, which are measured as number of nucleotide bases 

-between genomic regions (aligned to the same reference genome) and computed 

-using arithmetic operations between region coordinates.

-GMQL is a high-level, declarative language that allows expressing queries 

-easily over genomic regions and their metadata, in a way similar to what can 

-be done with the Structured Query Language (SQL) over a relational database.

-GMQL approach exhibits two main differences with respect to other tools 

-based on Hadoop, mapReduce framework, and Spark engine technologies 

-to address similar biomedical problems:\newline

-

-* GMQL:

-

-    1. reads from processed datasets

-    2. supports metadata management

-    

-* Others:

-

-    1. read generally from raw or alligned data from NGS machines

-    2. provide no support for metadata management

-

-GMQL is the appropriate tool for querying numerous processed genomic datasets 

-and very many samples that are becoming available.

-Note however that GMQL performs worse than some other available systems on a 

-small number of small-scale datasets, but these other systems are not 

-cloud-based; hence, they are not adequate for efficient big data processing 

-and, in some cases, they are inherently limited in their 

-data management capacity, as they only work as RAM memory resident processes.

-

-## Query structure

-

-A GMQL operation is expressed as a sequence of GMQL operations with the 

-following structure:

-$$< variable > = operator(< parameters >) < variable >;$$

-where each $< variable >$ stands for a GDM dataset

-

-This RGMQL package brings GMQL functionalities into R environemnt, 

-allowing users to build directly a GMQL query without knowing the GMQL syntax.

-In RGMQL every GMQL operations is translated into a R function 

-and expressed as:

-$$ variable = operator(variable, parameters)$$

-

-It is very similar to the GMQL syntax for operation expression although 

-expressed with the R idiomatic paradigm and logic, with parameters totaly 

-builded using R native data structures such as lists, matrices, 

-vectors or R logic conditions.

-

-

-# Processing Environments

-

-In this section, we show how GMQL processing is built in R, which operations 

-are available in RGMQL, and the difference beetween local 

-and remote dataset processing.

-

-## Local Processing

-

-RGMQL local processing consumes computational power directly from local 

-CPUs/system while managing datasets (both GMQL or generic text plain datasets).

-

-### Initialization

-

-Load and attach the GMQL package in a R session using library function:

-

-```r

-library('RGMQL')

-```

-Before starting using any GMQL operation we need to initialise the GMQL 

-context with the following code:

-

-```r

-init_gmql()

-```

-The function *init_gmql()* initializes the context of scalable data management 

-engine laid upon Spark and Hadoop.

-Details on this and all other functions are provided in the R documentation 

-for this package (e.g., help(RGMQL)).

-

-### Read Dataset

-

-After initialization we need to read datasets.

-We already defined above the formal definition of dataset and the power of 

-GMQL to deal with data in a variety of standard tab-delimited text formats.

-In the following, we show how to get data from different sources.\newline

-We distinguish two different cases:

-

-1. Local dataset:\newline

-A local dataset is a folder with sample files (region files and correspondent 

-metadata files) on the user computer.

-As data are already in the user computer, we simply execute:

-

-

-```r

-gmql_dataset_path <- system.file("example", "EXON", package = "RGMQL")

-data_out = read_dataset(gmql_dataset_path)

-```

-In this case we are reading a dataset named EXON specified by path.

-It doens't matter what kind of format the data are, *read_dataset()* read many 

-standard tab-delimited text formats without specified any paramter at input.

-

-2. GRangesList:\newline

-For better integration in the R environment and with other packages, 

-we provide a *read()* function to read directly from R memory/environment 

-using GRangesList as input.

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-