Posted: May 13th, 2023

DSBD2 essay

See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/303031880
Big IoT and Social Networking Data for Smart Cities – Algorithmic
Improvements on Big Data Analysis in the Context of RADICAL City
Applications
Conference Paper · July 2016: 2024 – Do my homework – Help write my assignment online
DOI: 10.5220/0005934503960405
CITATIONS
17
READS
1,889
6 authors, including:
Some of the authors of this publication are also working on these related projects:
CloudPerfect View project
Smart Cities View project
Evangelos Psomakelis
National Technical University of Athens
12 PUBLICATIONS 51 CITATIONS
SEE PROFILE
Fotis Aisopos
National Technical University of Athens
31 PUBLICATIONS 260 CITATIONS
SEE PROFILE
Antonios Litke
National Technical University of Athens
48 PUBLICATIONS 464 CITATIONS
SEE PROFILE
Konstantinos Tserpes
Harokopio University
112 PUBLICATIONS 872 CITATIONS
SEE PROFILE
All content following this page was uploaded by Konstantinos Tserpes on 06 June 2016: 2024 – Do my homework – Help write my assignment online.
The user has requested enhancement of the downloaded file.
BIG IOT AND SOCIAL NETWORKING DATA FOR SMART
CITIES:
Algorithmic improvements on Big Data Analysis in the context of RADICAL city
applications
Evangelos Psomakelis12
,Fotis Aisopos1
, Antonios Litke1
, Konstantinos Tserpes21
, Magdalini
Kardara1
and Pablo Martínez Campo3
1Distributed Knowledge and Media Systems Group, National Technical University of Athens, Zografou Campus, Athens,
Greece
2
Informatics and Telematics Dept, Harokopio University of Athens, Greece
3Communications Engineering department, University of Cantabria, Santander, Spain
{fotais, litke, nkardara, tserpes, vpsomak}@mail.ntua.gr,pmartinez@tlmat.unican.es
Keywords: Internet of Things, Social Networking, Big Data Aggregation and Analysis, Smart City applications,
Sentiment Analysis, Machine Learning
Abstract: In this paper we present a SOA (Service Oriented Architecture)-based platform, enabling the retrieval and
analysis of big datasets stemming from social networking (SN) sites and Internet of Things (IoT) devices,
collected by smart city applications and socially-aware data aggregation services. A large set of city
applications in the areas of Participating Urbanism, Augmented Reality and Sound-Mapping throughout
participating cities is being applied, resulting into produced sets of millions of user-generated events and
online SN reports fed into the RADICAL platform. Moreover, we study the application of data analytics
such as sentiment analysis to the combined IoT and SN data saved into an SQL database, further
investigating algorithmic and configurations to minimize delays in dataset processing and results retrieval.
1 INTRODUCTION
Modern cities are increasingly turning towards
ICT technology for confronting pressures associated
with demographic changes, urbanization, climate
change (Romero Lankao, 2008 – Affordable Custom Essay Writing Service | Write My Essay from Pro Writers) and globalization.
Therefore, most cities have undertaken significant
investments during the last decade in ICT
infrastructure including computers, broadband
connectivity and recently sensing infrastructures.
These infrastructures have empowered a number of
innovative services in areas such as participatory
sensing, urban logistics and ambient assisted living.
Such services have been extensively deployed in
several cities, thereby demonstrating the potential
benefits of ICT infrastructures for businesses and the
citizens themselves. During the last few years we
have also witnessed an explosion of sensor
deployments and social networking services, along
with the emergence of social networking (Conti et
al., 2011) and internet‐of‐things technologies (Perera
et al., 2013; Sundmaeker et al., 2010 – Essay Writing Service: Write My Essay by Top-Notch Writer) Social and
sensor networks can be combined in order to offer a
variety of added‐value services for smart cities, as
has already been demonstrated by various early
internet‐of‐things applications (such as
WikiCity(Calabrese et al., 2007), CitySense(Murty
et al., 2007), GoogleLatitude(Page and Kobsa,
2010 – Essay Writing Service: Write My Essay by Top-Notch Writer)), as well as applications combining social and
sensor networks (as for example provided by
(Breslin and Decker, 2007; Breslin et al., 2009) and
(Miluzzo et al., 2007). Recently, the benefits of
social networking and internet‐of‐things
deployments for smart cities have also been
demonstrated in the context of a range of EC
co‐funded projects (Hernández-Muñoz et al., 2011;
Sanchez, 2010 – Essay Writing Service: Write My Essay by Top-Notch Writer).
Current Smart City Data Analysis implies a wide
set of activities aiming to turn into actionable data
the outcome of complex analytics processes. This
analysis comprises among others: i) analysis of
thousands of traffic, pollution, weather, waste,
energy and event sensory data to provide better
services to the citizens, ii) event and incident
analysis using near real-time data collected by
citizens and devices sensors, iii) turning social
media data related to city issues into event and
sentiment analysis , and many others. Combining
data from physical (sensors/devices) and social
sources (social networks) can give more complete,
complementary data and contributes to better
analysis and insights. In overall, smart cities are
complex social systems and large scale data
analytics can contribute into their sustainability,
efficient operation and welfare of the citizens.
Motivated by the modern challenges in smart
cities, the RADICAL approach (RADICAL, 2016: 2024 – Do my homework – Help write my assignment online)
opens new horizons in the development, deployment
and operation of interoperable social networking and
Internet of Things services in smart cities, notably
services that could be flexibly and successfully
customized and replicated across multiple cities. Its
main goal is to provide the means for cities and
SMEs to rapidly develop, deploy, replicate, and
evaluate a diverse set of sustainable ICT services
that leverage established IoT and SN infrastructures.
Application services deployed and pilotedinvolve: i)
Cycling Safety Improvement, ii) Products Carbon
Footprint Get research paper samples and course-specific study resources under   homework for you course hero writing service – Manage ment, iii) Object‐driven Data
Journalism, iv) Participatory Urbanism, v)
Augmented Reality, vi) Eco‐ consciousness, vii)
Sound map of a city and viii) City-R-Us: a
crowdsourcing app for collecting movement
information using citizens smartphones.
The RADICAL platform is an open platform
having as added value the capability to easily
replicate the services in other smart cities, the ability
to co-design services with the involvement of cities’
Living Labs, and the use of added value services that
deal with the application development, the
sustainability analysis and the governance of the
services.
The RADICAL approach emphasizes on the
sustainability of the services deployed, targeting
both environmental sustainability and business
viability. Relevant indicators (e.g., CO2 emissions,
Citizens Satisfaction) are established and monitored
as part of the platform evaluation.End users
(citizens) in modern smart cities are increasingly
looking for media‐rich services offered under
different space‐, context‐, and situational conditions.
The active participation and interaction of citizens
can be a key enabler for successful and sustainable
service deployments in future cities. Social networks
hold the promise to boost such participation and
interaction, thereby boosting participatory connected
governance within the cities. However, in order to
enable smart cities get insight information on how
citizens think, act and talk about their city it is
important to understand their opinion and sentiment
polarity on issues related to their city context. This is
where sentiment analysis can play a significant role.
As social media data bring in significant Big Data
challenges (especially for unstructured data streams)
it will be important to find effective ways to analyse
sentimentally those data for extracting value
information and within specific time windows.
This paper has the following contributions:
 Innovative smart city infrastructure for
uniform social and IoT big data aggregation
and combination.
 Comperative study over Sentiment Analysis
techniques efficiency, to reduce record,
retrieval, update and processing time.
 A novel technique for n-grams storage and
frequency representation in the context of
big data Sentiment Analysis.
The rest of the paper is structured as follows:
Section 2 gives an overview of related and similar
works that can be found in the international
literature and in projects funded by the European
Commission. Section 3 presents the RADICAL
architecture and approach. Section 4 presents details
about the Sentiment Analysis problem and related
experiments, while in section 5 we provide the
future work to be planned in the context of
RADICAL and the conclusions we have come into.
2 RELATED WORK
Recently, various analytical services such as
sentiment analysis found their way into Internet of
Things (IoT) applications. With the devices that are
able to convey human messages over the internet
meeting an exponential growth, the challenge now
revolves around big data issues. Traditional
approaches do not cope with the requirements posed
from applications for analytics in e.g. high velocity
rates or data volumes. As a result, the integration of
IoT with social sensor data put common tasks like
feature extraction, algorithm training or model
updating to the test.
Most of the algorithms are memory-resident and
assume a small data size (He et al., 2010 – Essay Writing Service: Write My Essay by Top-Notch Writer) and once
this threshold is exceeded, the algorithms’ accuracy
and performance degrades to the point they are
useless. Therefore even if we focused solely on
volume challenges, it is intuitively expected that the
accuracy of the supervised algorithms will be
affected. An attempt from (Liu et al., 2013) to use
Naïve Bayes in an increasingly large data volume,
showed that a rapid fall of the algorithms accuracy is
followed by a continuous, smooth increase
asymptotically tending from the lower end to the
baseline (best accuracy under normal data load).
Rather than testing the algorithm’s limitations,
most of the other approaches are focusing on
implementing parallel and distributed versions of the
algorithms such as (He et al., 2010 – Essay Writing Service: Write My Essay by Top-Notch Writer; Read et al.,
2015 – Research Paper Writing Help Service). In fact most of them rely on the Map-Reduce
framework so as to achieve high throughput
classification (Amati et al., 2014: 2024 – Essay Writing Service | Write My Essay For Me Without Delay; Sakaki et al.,
2013; Wang et al., 2014: 2024 – Essay Writing Service. Custom Essay Services Cheap; Zhao et al., 2014: 2024 – Essay Writing Service. Custom Essay Services Cheap) whereas
a number of toolkits have been presented with
implementations of distributed or parallel versions
of machine learning algorithms such as (“Apache
Mahout: Scalable machine learning and data
mining,” n.d., “MEKA: A Multi-label Extension to
WEKA,” n.d.; Bifet et al., 2010 – Essay Writing Service: Write My Essay by Top-Notch Writer). While these
solutions put most of the emphasis in the model and
the optimization of the classification task in terms of
accuracy and throughput, there is a rather small body
of research dealing with the problem of feature
extraction in high pace streams. The standard
solution that is considered is the use of a sliding
window and the application of standard feature
extraction techniques in this small set. In cases
where the stream’s distribution is variable, a sliding
window kappa-based measure has been proposed
(Bifet and Frank, 2010 – Essay Writing Service: Write My Essay by Top-Notch Writer).
As reported in (Strohbach et al., 2015 – Research Paper Writing Help Service), another
domain of intense research in the area of scalable
analytics is for an architecture that combines both
batch and stream processing over social and IoT data
while at the same time considering a single model
for different types of documents (e.g. tweets Vs
blogposts). Sentiment analysis is a typical task that
requires batch modeling in order to generate the
golden standards for each of the classes. This
process is also the most computationally intense, as
the classification task itself is usually a CPU bound
task (i.e. run the classification function). In a data
streaming scenario the golden standards must be
updated in a batch mode, whereas the feature
extraction and classification must take place in real
time.
Perhaps the most prominent example of such an
architecture is the Lambda Architecture (Marz and
Warren, 2015 – Research Paper Writing Help Service) pattern which solves the problem of
computing arbitrary functions on arbitrary data in
realtime by combining a batch layer for processing
large scale historical data and a streaming layer for
processing items being retrieved in real time from an
input queue or analytics in e.g. high velocity rates or
data volumes. As a result, the integration of IoT with
social sensor data put common tasks like feature
extraction, algorithm training or model updating to
the test.Most of the algorithms are memory-resident
and assume a small data size (He et al., 2010 – Essay Writing Service: Write My Essay by Top-Notch Writer) and
once this threshold is exceeded, the algorithms’
accuracy and performance degrades to the point they
are useless. Therefore even if we focused solely on
volume challenges, it is intuitively expected that the
accuracy of the supervised algorithms will be
affected. An attempt from Liu et al (Liu et al., 2013)
to use Naïve Bayes in an increasingly large data
volume, showed that a rapid fall of the algorithms
accuracy is followed by a continuous, smooth
increase asymptotically tending from the lower end
to the baseline (best accuracy under normal data
load).
3. THE RADICAL APPROACH
The RADICAL platform integrates components
and tools from (SocIoS, 2013) and (SmartSantander,
2013) projects, in order to support innovative smart
city services, leveraging information stemming from
Social Networks (SN) and Internet of Things
devices. Using the aforementioned tools, it can
collect, combine, analyze, process, visualize and
provide uniform access to big datasets of Social
Network content (e.g. tweets) and Internet of Things
information (e.g. sensor measurements or citizen
smartphone reports).
The architecture of the RADICAL platform is
depicted in Figure 1. As can be observed, all IoT
data are pushed into the platform through the
respective Application Programming Interfaces (IoT
API and Repository API) and are forwarded to the
RADICAL Repository, comprised by a MySQL
database, formed based on the RADICAL Object
Model. The device-related data, as dictated by this
object model, are saved in the form of Observations
and Measurements. Observations correspond to
general IoT events reported (e.g. a sensor report or
bicycle “check-in” event), while Measurements to
more specific metrics included in an Observation
(e.g. Ozone measurements (mpcc) or bicycle current
speed (km/h)). On the other hand, SN data are
accessed in real time from the underlying SN
adaptors, by communicating with the respective
Networks’ APIs. In cases of Social Networks like
Foursquare that provide plain venues and statistics,
the adaptor-like data structures do not make sense,
thus relevant Social Enablers are used to retrieve
venue-related information data.
On top of the main platform, RADICAL
delivers a set of tools (Application Get research paper samples and course-specific study resources under   homework for you course hero writing service – Manage ment
layer) that allow end users to make better use of the
RADICAL platform, such as configuring the
registered IoT devices or extracting general activity
statistics, through the RADICAL Configuration API.
Lastly, the RADICAL Data API allows smart city
services to access the different sources of
information (social networks, IoT infrastructures,
city applications), combine data and perform data
analysis by using the appropriate platform tools.
Figure 1: RADICAL Platform Architecture
As can be seen in the Service Application Layer,
in the context of RADICAL a wide range of Smart
City services of various scopes has been developed:
 Citizen journalism and Participatory
Urbanism: Those two interrelated services
allow citizens reporting events ofinterest in
the city, by posting images, text and metadata
through their smartphones.
 Cycling Safety:Cyclists, acting as human
sensors can report the situation in the city
streets through their smartphones.
 Monitoring the carbon footprint of
products, people and services: By using a
range of sensors, the CO2 emissions in
specific places in a city may be monitored.
 Augmented Reality in Points of Interest
(POI): Tourists use their smartphones to
identify and receive information about points
of interest in a city.
 Propagation of eco-consciousness:Leverages
on the viral effect in the propagation of
information in the social networks as well as
the recycling policy of a city, through
monitoring and reporting relevant actions on
citizens’ smartphones.
 Social-Orientated Urban Noise Decibel
Measurement Application: Noise sensors are
employed throughout the city and citizens are
able to report and comment noise-related
information through SNs under a hashtag.
 City Reporting application for the use of
Urban Services: This service gathers sensory
data along with SN check-ins in city venues,
to construct a traffic map throughout the city,
leveraging the process load of anycentralized
decision making process.
The aforementioned services are piloted in six
European participating cities: Aarhus, Athens,
Genoa, Issy les Moulineaux, Santander and the
region of Cantabria. Figure 2illustrates a screenshot
example of the RADICAL Cities’ Dashboard, where
RADICAL
Data
Repository
City
Applications
Radical Data API
Service Application Layer
Sound MAPP
Municipality
Cockpit
City R-U S
Service
aggregator
Repository
Gateway
Application Get research paper samples and course-specific study resources under   homework for you course hero writing service – Manage ment Tools
Governance
Toolkit
Application
Development
Toolkit
Sustainability
services
Resource
Directory
Service
Storer
IoT Get research paper samples and course-specific study resources under   homework for you course hero writing service – Manage r Event Broker
Node
Get research paper samples and course-specific study resources under   homework for you course hero writing service – Manage r
Data Rep.
Configurator
Radical Configuration API
IoT devices
Repository API
Configuration Tools
Social Media
…
Register
Get research paper samples and course-specific study resources under   homework for you course hero writing service – Manage r
IOT API SN Adaptors
SocIoS Core Service
SN Enablers
Event
Detection
Sentiment
Analysis
GWFPSens GW4IoT Devices … GW4Serv
Platform Tools
Carbon
Footprint
Augmented
Reality
EcoConciousness
Object Driven
Journalism
Participatory
Urbanism
Cycling
Safety
Social Networks –
Venues Services
general statistics on device registration and activity
for a service throughout different cities in a specific
time period is provided. In overall, during the last
pilot iteration, RADICAL Repository had captured a
total of 5.636 active IoT devices sending 728.253
Observations and 5.461.776 Measurements.
Most of the services above depend on the
aggregation of those IoTdata with social data
stemming from online Social Network sites. E.g. in
the Participatory Urbanism service, citizens’ reports
sent through smartphones and saved in RADICAL
Repository are combined with relevant tweets (under
a city service hashtag), as well as POI information
that can be collected from similar SNs.
Figure 2: RADICAL Cities Dashboard presents smartphone registrations and measurements for the AR service in the cities
of Santander and Cantabria over a period
Thus, given the size of the datasets acquired by
smart city services, along with the rich social media
content that can be retrieved through the RADICAL
platform adaptors, big data aggregation and analysis
challenges arise. Data Analysis tools are the ones
that further process the data in order to provide
meaningful results to the end user, i.e. Event
Detection or Sentiment Analysis.
When it comes to Big Data, as in the RADICAL
case, where millions of user-reported events are
aggregated along with millions of SN posts and an
extraction of general results is required, the
challenge accrued is two-fold: First, the tool must
ensure the accuracy of the analysis, in the sense that
data classification is correct to a certain and
satisfactory extent, and second, processing time
must be kept under certain limits, so that results
retrieval process delay is tolerable by end-users.
Moreover, it is apparent in such analysis that a tradeoff between effectiveness and efficiency exists. The
latter is a most crucial issue in Big Data analysis and
apart from the policy followed in data querying (e.g.
for queries preformed in an SQL database), it is also
related to the algorithmic techniques employed
foranalyzing those datasets.
In the context of this work, we focus on the
Sentiment Analysis on the big IoT and SN related
datasets of RADICAL, as this was the most popular
functionality among participating cities and almost
all of the RADICAL Smart City services presented
above make use of it. The goal of the Sentiment
Analysis service is to extract sentiment expressive
patterns from user-generated content in social
networks or IoT-originated text posts. The service
comes to the aid of the RADICAL city
administrators, helping them to categorize polarized
posts, meaning sentimentally charged text, e.g.
analyse citizens’ posts to separate subjective from
objective opinions or count the overall positive and
negative feedback, concerning a specific topic or
event in the city.
4. SENTIMENT ANALYSIS
EXPERIMENTS AND
PERFORMANCE IMPROVEMENT
4.1 Introduction
The term Sentiment Analysis refers to an
automatic classification problem. Its techniques are
trying to distinguish between sentences of natural
language conveying positive (e.g. happiness, pride,
joy), negative (e.g. anger, sadness, jealously) or even
neutral (no sentiment texts like statements, news,
reports)emotion (called sentiment for our purposes)
(Pang et al., 2002).
A human being is capable of understanding a
great variety of emotions from textual data. This
process of understanding is based on complicated
learning procedures that we all go through while
using our language as a means of communication, be
it actively or passively. It requires imagination and
subjectivity in order to fully understand the meaning
and hidden connections of each word in a sentence,
two things that machines lack.
The most common practice is to extract
numerical features out of the natural language
(Godbole et al., 2007). This process translates this
complex means of communication into something
the machine can process.
4.2 Natural Language Processing
In order to process the natural language data, the
computer has to take some pre-processing actions.
These actions include the cleansing of irrelevant,
erroneous or redundant data and the transformation
of the remaining data in a form more easily
processed.
Cleansing the data has become a subjective task,
depending on the purposes of each researcher and
the chosen machine learning algorithms. The
transformation of the sentences in another form now
is clearly studied and each approach has some
advantages and disadvantages. This paper will detail
three approaches, two widely used and one that had
some success in improving the accuracy of the
algorithms: the bag of word, N-Gramsand N-Gram
Graphs(Aisopos et al., 2014: 2024 – Essay Writing Service. Custom Essay Services Cheap; Fan and Khademi,
2014: 2024 – Essay Writing Service | Write My Essay For Me Without Delay; Giannakopoulos et al., 2008 – Affordable Custom Essay Writing Service | Write My Essay from Pro Writers; Pang and Lee,
2008 – Affordable Custom Essay Writing Service | Write My Essay from Pro Writers).
The bag of words approach is perhaps the most
simple and common one. It regards each sentence as
a set of words, disregarding their grammatical
connections and neighbouring relations. It splits
each sentence based on the space character (in most
languages) and then forms a set of unrelated words
(a bag of words as it is commonly called). Then each
word in this bag can be disregarded or rated by a
numerical value, in order to create a set of numbers
instead of words.
The N-Grams are a bit more complex. They also
form a bag of words but now each sentence is split
into pseudo-words of equal length. A sliding
window of N characters is rolling on the sentence
creating this bag of pseudo-words. For example if
N=3 the sentence “This is a nice weather we have
today!” will be split in the bag {‘Thi’, ‘his’, ‘is ’, ‘s
i’, ‘ is’, ‘is ’, ‘s a’, ‘ a ’, ‘a n’, ‘ ni’, ‘nic’, ‘ice’, ‘ce ’,
‘e w’, ‘ we’, ‘wea’, ‘eat’, ‘ath’, ‘the’, ‘her’, ‘er ’, ‘r
w’, ‘ we’, ‘we ’, ‘e h’, ‘ ha’, ‘hav’, ‘ave’, ‘ve ’, ‘e t’,
‘ to’, ‘tod’, ‘oda’, ‘day’, ‘ay!’}.
This technique takes into regard the direct
neighbouring relations by creating a continuous
stream of words, it still ignores the indirect relations
between words and even the relations between the
produced N-Grams. Of course it is impossible to
have a predefined set of numerical ratings for each
one of these pseudo-words because each sentence
and each N number (which is defined arbitrarily by
the researcher) produces a different set of pseudowords(Psomakelis et al., 2014: 2024 – Essay Writing Service | Write My Essay For Me Without Delay). So machine learning
is commonly used to replace these words with
numerical values and create sets of numbers which
can be aggregated to sentence level.
An improvement on that approach aims to take
into consideration the neighbouring relations
between the produced N-Grams. This approach is
called N-Gram Graphs and its main concept is to
create a graph connecting each N-Gram with its
neighbours in the original sentence. So each node in
this graph is an N-Gram and each edge is a
neighbouring relation(Giannakopoulos et al., 2008 – Affordable Custom Essay Writing Service | Write My Essay from Pro Writers).
This approach gives a variety of new informationto
the researchers and to the machine learning
algorithms, including information about the context
of words, making it a clear improvement of the
simple N-Grams(Aisopos et al., 2014: 2024 – Essay Writing Service. Custom Essay Services Cheap). The only
drawbacks are the complexity it adds to the process
and the difficulties of storing, accessing and
updating a graph of textual data.
4.3 Dataset Improvements
At the core of sentiment analysis is its dataset.
We are gathering and employing bigger and bigger
datasets in order to better train the algorithms to
distinguish what is positive and what is negative.
Classic storage techniques are proving more and
more cumbersome for large datasets. ArrayLists and
most Collections are adding a big overhead to the
data so they are not only enlarging the space
requirements for its storage but they are also
delaying the analysis process. So new techniques for
data storage and retrieval are needed, techniques that
will enable us to store even bigger datasets and
access them with even smaller delays.
The most commonly used such technique is the
Hash List(Fan and Khademi, 2014: 2024 – Essay Writing Service | Write My Essay For Me Without Delay), which first
hashes the data in a certain, predefined amount of
buckets and then creates a List in each bucket to
resolve any collisions. This method’s performance is
heavily dependent on the quality of the hash
function and its ability to equally split the data into
the buckets. The target is to have as small lists as
possible. That is the case because finding the right
bucket for a certain piece of data is done in O(1)
time but looking through the List in that bucket for
the correct spot to store the piece of data is done in
O(n) time where n is the number of data pieces in
the List.
Moreover, in Java which is the programming
language that we are using, each List is an object
containing one object for each data piece. All these
objects create an overhead that is not to be ignored.
In detail the estimated size that a hash list will
occupy is calculated as:
12 + ((B − E) ∗ 12) + (E ∗ 4)
+ (U ∗ (N ∗ 2 + 72))
Equation 1: Size estimation of Hash List where N=NGram
Length, U=Unique NGrams, B=Bucket Size, E=Empty
Buckets.
The worst case for storage but best for access
time is when almost each data piece has its own
bucket. In this case, for N=5, S=11881376, U=S,
B=(26^N)*2, E=11914220, we have a storage size
of 1110 MB. The best case for storage but worse for
access time is when all data pieces are in a small
number of buckets, in big lists. In this case for N=5,
S=11881376, U=1, B=(26^N)/2, E=2006 – Write a paper; Professional research paper writing service – Best essay writers10 we have
a storage size of 23 MB. In an average case of N=5,
S=11881376, U=7510766, B=26^N, E=2679046 we
have 682 MB of storage space needed. The sample
for the above examples was the complete range of 5-
Grams for the 26 lowercase English characters
which are 26^5 = 11881376.
Our proposed technique now, the one that we
call Dimensional Mapping, has a standard storage
space, depending only on the length of the N-Grams.
The idea is to store only the weight of each N-Gram
with the N-Gram itself being the pointer to where it
is stored. That is achieved by creating an Ndimensional array of integers where each character
of the N-Gram is used as an index. So, in order to
access the weight of the 5-Gram ‘fdsgh’ in the table
DM we would just read the value in cell
DM[‘f’][‘d’][‘s’][‘g’][‘h’]. A very simple mapping
is used between the characters and an integers: after
a very strict cleansing process where we convert all
characters in lowercase and discard all characters but
the 26 in the English alphabet, we are just
subtracting the ASCII value of ‘a’. Due to the serial
nature of the characters that gives us an integer
between 0 and 26 that we can use as an index. A
more complex mapping can be used in order to
include more characters or even punctuation that we
now ignore.
The Dimensional Mapping has a standard storage
size requirement, dependent only on the length of
the N-Grams as we mentioned before. The size it
occupies can be estimated by the following formula:
(26 ) ∗ 4 + ∑((26 −
) ∗ 12)
=
=1
Equation 2: Dimensional Mapping size estimation with N
being the length of N-Grams.
This may seem large but for the 5-Grams the
estimated size is just 51 MB. Compared to the worst
case of Hash Lists (1110 MB) or even the average
case (682 MB) it seems like a huge improvement.
This is caused due to the fact that the
multidimensional array stores primitive values and
not objects, which reduces the overhead greatly.
Moreover, we can now say that accessing and
updating a certain data piece can be done in O(1)
time with absolute certainty, with no dependency on
the data itself or a hash function. This had
significant results in speeding up the execution times
of the analysis, enabling us to look into streaming
data and semi-supervised machine learning
algorithms.
4.4 Results
We measured three main KPIs for the result
comparison. Two of them (success ratio, kappa
variable) were measuring the success ratio of
classification and one (execution time) the
algorithmic improvement. We present them bellow.
We run experiments on 5-Grams stored in classic
ArrayList format, in Hash Lists and in Dimensional
Mapping. After storing the N-Grams in these
formats we applied a 10-fold cross validation on
each one of the seven machine learning algorithms
we chose: Naïve Bayesian Networks, C4.5, Support
Vector Machines, Logistic Regression, Multilayer
Perceptrons, Best-First Trees and Functional Trees.
Then we recorded the three KPIs for each one of
these 21 experiments. The results for the first two
KPIs are shown in the bar chart that follows. In the
same chart we have included the KPIs for a
threshold based classification, using an arbitrarily set
threshold.
Figure 3: A comparison of the three KPIs as shown in the
sentiment analysis experiments
As of the execution times the following table
contains a summary of the results:
Table 1: Execution time in seconds summary – comparing
for the various algorithms and techniques
ArrayLists Hash
List
Dimensional
Mapping
Thresholds 1691 5 4
Naïve Bayes 12302 7 7
C4.5 21535 9 8
SVM 20662 147 177
Logistic
Regression
22251 9 11
MLP 21224 41 48
BFTree 23319 25 19
FTree 22539 16 16
5. CONCLUSIONS
RADICAL platform, as presented in the current
work, successfully combines citizens’ posts retrieved
through smartphone applications and Social
Networks in the context of smart city applications, to
produce a testbed for applying multiple analysis
functionalities and techniques. The exploitation of
resulting big aggregated datasets pose multiple
challenges, with timely-efficient analysis being the
most important. Focusing on data storage and
representation, multiple techniques were examined
in the experiments performed, in order to come up
with the optimal algorithmic approach of
Dimensional Mapping. In the future the authors plan
to use even larger and more complex datasets,
further leveraging on the effectiveness of these
social networking services.
ACKNOWNLEDGEMENTS
The authors would like to acknowledge the
RADICAL consortium for their collaboration during
the research project. This work was supported by the
European Union’s Competitiveness and Innovation
Framework Programme under grant agreement no
325138.
REFERENCES
Aisopos, F., 0001, G.P., Tserpes, K., Varvarigou,
T.A., 2014: 2024 – Essay Writing Service. Custom Essay Services Cheap. Content vs. context for
sentiment analysis: a comparative analysis
over microblogs., in: Munson, E.V.,
Strohmaier, M. (Eds.), HT. ACM, pp. 187–
196.
Amati, G., Angelini, S., Bianchi, M., Costantini, L.,
Marcone, G., 2014: 2024 – Essay Writing Service | Write My Essay For Me Without Delay. A scalable approach to
near real-time sentiment analysis on social
networks. Inf. Filter. Retr. 12.
Apache Mahout: Scalable machine learning and data
mining [WWW Document], n.d. URL
http://mahout.apache.org/ (accessed
4.8.15).
Bifet, A., Frank, E., 2010 – Essay Writing Service: Write My Essay by Top-Notch Writer. Sentiment Knowledge
Discovery in Twitter Streaming Data, in:
Proceedings of the 13th International
Conference on Discovery Science, DS’10.
Springer-Verlag, Berlin, Heidelberg, pp. 1–
15.
Bifet, A., Holmes, G., Kirkby, R., Pfahringer, B.,
2010 – Essay Writing Service: Write My Essay by Top-Notch Writer. MOA: Massive Online Analysis. J.
Mach. Learn. Res. 11, 1601–1604.
Breslin, J., Decker, S., 2007. The Future of Social
Networks on the Internet: The Need for
Semantics. IEEE Internet Comput. 11, 86–
90. doi:10.1109/MIC.2007.138
Breslin, J.G., Decker, S., Hauswirth, M., Hynes, G.,
Phuoc, D.L., Passant, A., Polleres, A.,
Rabsch, C., Reynolds, V., 2009. Integrating
Social Networks and Sensor Networks, in:
Proceedings on the W3C Workshop on the
Future of Social Networking.
Calabrese, F., Kloeckl, K., Ratti, C., 2007. Wikicity:
Real-time location-sensitive tools for the
city IEEE Pervasive Computing, 390–413.
Conti, M., Passarella, A., Pezzoni, F., 2011. A
model for the generation of social network
graphs., in: WOWMOM. IEEE Computer
Society, pp. 1–6.
Fan, M., Khademi, M., 2014: 2024 – Essay Writing Service | Write My Essay For Me Without Delay. Predicting a Business
Star in Yelp from Its Reviews Text Alone.
CoRR abs/1401.0864.
Giannakopoulos, G., Karkaletsis, V., Vouros, G.A.,
Stamatopoulos, P., 2008 – Affordable Custom Essay Writing Service | Write My Essay from Pro Writers. Summarization
system evaluation revisited: N-gram
graphs. TSLP 5.
Godbole, N., Srinivasaiah, M., Skiena, S., 2007.
Large-Scale Sentiment Analysis for News
and Blogs, in: Proceedings of the
International Conference on Weblogs and
Social Media (ICWSM).
GrowSmarter, 2016: 2024 – Do my homework – Help write my assignment online. Grow Smarter project [WWW
Document]. URL http://www.growsmarter.eu/home/
Gubbi, J., Buyya, R., Marusic, S., Palaniswami, M.,
2014: 2024 – Essay Writing Service. Custom Essay Services Cheap. Internet of Things (IoT): A Vision,
Architectural Elements, and Future
Directions. CoRR abs/1207.0203.
He, Q., Zhuang, F., Li, J., Shi, Z., 2010 – Essay Writing Service: Write My Essay by Top-Notch Writer. Parallel
Implementation of Classification
Algorithms Based on MapReduce, in: Yu,
J., Greco, S., Lingras, P., Wang, G.,
Skowron, A. (Eds.), Rough Set and
Knowledge Technology, Lecture Notes in
Computer Science. Springer Berlin
Heidelberg, pp. 655–662.
Hernández-Muñoz, J.M., Vercher, J.B., Muñoz, L.,
Galache, J.A., Presser, M., Gómez, L.A.H.,
Pettersson, J., 2011. Smart Cities at the
Forefront of the Future Internet., in:
Domingue, J., Galis, A., Gavras, A.,
Zahariadis, T.B., Lambert, D., Cleary, F.,
Daras, P., Krco, S., Müller, H., Li, M.-S.,
Schaffers, H., Lotz, V., Alvarez, F., Stiller,
B., Karnouskos, S., Avessta, S., Nilsson,
M. (Eds.), Future Internet Assembly,
Lecture Notes in Computer Science.
Springer, pp. 447–462.
Liu, B., Blasch, E., Chen, Y., Shen, D., Chen, G.,
2013. Scalable sentiment classification for
Big Data analysis using Na #x00EF;ve
Bayes Classifier, in: Big Data, 2013 IEEE
International Conference on. pp. 99–104.
doi:10.1109/BigData.2013.6691740
Marz, N., Warren, J., 2015 – Research Paper Writing Help Service. Big Data: Principles and
best practices of scalable realtime data
systems, 1 edition. ed. Manning
Publications, Westampton.
MEKA: A Multi-label Extension to WEKA [WWW
Document], n.d. URL
http://meka.sourceforge.net/ (accessed
3.27.15).
Miluzzo, E., Lane, N.D., Eisenman, S.B., Campbell,
A.T., 2007. CenceMe – Injecting Sensing
Presence into Social Networking
Applications., in: Kortuem, G., Finney, J.,
Lea, R., Sundramoorthy, V. (Eds.),
EuroSSC, Lecture Notes in Computer
Science. Springer, pp. 1–28.
Murty, R., Gosain, A., Tierney, M., Brody, A.,
Fahad, A., Bers, J., Welsh, M., 2007.
Harvard University Technical Report.
TR‐13‐07.
Page, X.W., Kobsa, A., 2010 – Essay Writing Service: Write My Essay by Top-Notch Writer. Navigating the social
terrain with google latitude. iConference
Urbana‐Champaign, p.174‐178.
Pang, B., Lee, L., 2008 – Affordable Custom Essay Writing Service | Write My Essay from Pro Writers. Opinion mining and
sentiment analysis. Found. Trends Inf. Retr.
2, 1–135.
Pang, B., Lee, L., Vaithyanathan, S., 2002. Thumbs
up? Sentiment Classification Using
Machine Learning Techniques, in:
emnlp2002. Philadelphia, Pennsylvania, pp.
79–86.
Perera, C., Zaslavsky, A.B., Christen, P.,
Georgakopoulos, D., 2013. Sensing as a
Service Model for Smart Cities Supported
by Internet of Things. CoRR
abs/1307.8198.
Psomakelis, E., Tserpes, K., Anagnostopoulos, D.,
Theodora, V., 2014: 2024 – Essay Writing Service | Write My Essay For Me Without Delay. Comparing Methods
for Twitter Sentiment Analysis. Proc. 6th
Int. Conf. Knowl. Discov. Inf. Retr. – KDIR
2014: 2024 – Essay Writing Service | Write My Essay For Me Without Delay 225–232.
RADICAL, 2016: 2024 – Do my homework – Help write my assignment online. Rapid Deployment for Intelligent
Cities and Living, FP7 EU funded Research
project [WWW Document]. URL
http://www.radical-project.eu
Read, J., Martino, L., Olmos, P.M., Luengo, D.,
2015 – Research Paper Writing Help Service. Scalable multi-output label
prediction: From classifier chains to
classifier trellises. Pattern Recognit. 48,
2096–2109.
doi:10.1016/j.patcog.2015 – Research Paper Writing Help Service.01.004
Romero Lankao, P., 2008 – Affordable Custom Essay Writing Service | Write My Essay from Pro Writers. Urban areas and climate
change: Review of current issues and
trends. Issues Pap. 2011 Glob. Rep. Hum.
Settl.
Sakaki, T., Okazaki, M., Matsuo, Y., 2013. Tweet
analysis for real-time event detection and
earthquake reporting system development.
Knowl. Data Eng. IEEE Trans. On 25, 919–
931.
Sanchez, L., 2010 – Essay Writing Service: Write My Essay by Top-Notch Writer. SmartSantander: Experimenting
The Future Internet in the City of the
Future. Presented at the PIMRC2010 – Essay Writing Service: Write My Essay by Top-Notch Writer,
Istanbul, Turkey.
SmartSantander, 2013. SmartSantander, FP7 EU
funded Research project [WWW
Document]. URL
http://www.smartsantander.eu/
SocIoS, 2013. Exploiting Social Networks for
Building the Future Internet of Services,
FP7 EU funded Research project [WWW
Document]. URL
http://www.sociosproject.eu/
Strohbach, M., Ziekow, H., Gazis, V., Akiva, N.,
2015 – Research Paper Writing Help Service. Towards a big data analytics
framework for IoT and smart city
applications, in: Modeling and Processing
for Next-Generation Big-Data
Technologies. Springer, pp. 257–282.
Sundmaeker, H., Guillemin, P., Friess, P., Woelfflé,
S. (Eds.), 2010 – Essay Writing Service: Write My Essay by Top-Notch Writer. Vision and Challenges for
Realising the Internet of Things.
Publications Office of the European Union,
Luxembourg.
Wang, H., Can, D., Kazemzadeh, A., Bar, F.,
Narayanan, S., 2014: 2024 – Essay Writing Service. Custom Essay Services Cheap. A System for Realtime Twitter Sentiment Analysis of 2014: 2024 – Essay Writing Service. Custom Essay Services Cheap
U.S. Presidential Election Cycle., in: ACL
(System Demonstrations). The Association
for Computer Linguistics, pp. 115–120.
Zhao, J., Dong, L., Wu, J., Xu, K., 2014: 2024 – Essay Writing Service. Custom Essay Services Cheap. MoodLens:
an emoticon-based sentiment analysis
system for chinese tweets., in: 0001, Q.Y.,
Agarwal, D., Pei, J. (Eds.), KDD. ACM,
pp. 1528–1531.
View publication stats

Order | Check Discount